diff options
97 files changed, 4509 insertions, 390 deletions
diff --git a/Documentation/BreakingChanges.txt b/Documentation/BreakingChanges.txt new file mode 100644 index 0000000000..0532bfcf7f --- /dev/null +++ b/Documentation/BreakingChanges.txt @@ -0,0 +1,135 @@ += Upcoming breaking changes + +The Git project aims to ensure backwards compatibility to the best extent +possible. Minor releases will not break backwards compatibility unless there is +a very strong reason to do so, like for example a security vulnerability. + +Regardless of that, due to the age of the Git project, it is only natural to +accumulate a backlog of backwards-incompatible changes that will eventually be +required to keep the project aligned with a changing world. These changes fall +into several categories: + +* Changes to long established defaults. +* Concepts that have been replaced with a superior design. +* Concepts, commands, configuration or options that have been lacking in major + ways and that cannot be fixed and which will thus be removed without any + replacement. + +Explicitly not included in this list are fixes to minor bugs that may cause a +change in user-visible behavior. + +The Git project irregularly releases breaking versions that deliberately break +backwards compatibility with older versions. This is done to ensure that Git +remains relevant, safe and maintainable going forward. The release cadence of +breaking versions is typically measured in multiple years. We had the following +major breaking releases in the past: + +* Git 1.6.0, released in August 2008. +* Git 2.0, released in May 2014. + +We use <major>.<minor> release numbers these days, starting from Git 2.0. For +future releases, our plan is to increment <major> in the release number when we +make the next breaking release. Before Git 2.0, the release numbers were +1.<major>.<minor> with the intention to increment <major> for "usual" breaking +releases, reserving the jump to Git 2.0 for really large backward-compatibility +breaking changes. + +The intent of this document is to track upcoming deprecations for future +breaking releases. Furthermore, this document also tracks what will _not_ be +deprecated. This is done such that the outcome of discussions document both +when the discussion favors deprecation, but also when it rejects a deprecation. + +Items should have a clear summary of the reasons why we do or do not want to +make the described change that can be easily understood without having to read +the mailing list discussions. If there are alternatives to the changed feature, +those alternatives should be pointed out to our users. + +All items should be accompanied by references to relevant mailing list threads +where the deprecation was discussed. These references use message-IDs, which +can visited via + + https://lore.kernel.org/git/$message_id/ + +to see the message and its surrounding discussion. Such a reference is there to +make it easier for you to find how the project reached consensus on the +described item back then. + +This is a living document as the environment surrounding the project changes +over time. If circumstances change, an earlier decision to deprecate or change +something may need to be revisited from time to time. So do not take items on +this list to mean "it is settled, do not waste our time bringing it up again". + +== Git 3.0 + +The following subsections document upcoming breaking changes for Git 3.0. There +is no planned release date for this breaking version yet. + +Proposed changes and removals only include items which are "ready" to be done. +In other words, this is not supposed to be a wishlist of features that should +be changed to or replaced in case the alternative was implemented already. + +=== Changes + +* The default hash function for new repositories will be changed from "sha1" + to "sha256". SHA-1 has been deprecated by NIST in 2011 and is nowadays + recommended against in FIPS 140-2 and similar certifications. Furthermore, + there are practical attacks on SHA-1 that weaken its cryptographic properties: ++ + ** The SHAppening (2015). The first demonstration of a practical attack + against SHA-1 with 2^57 operations. + ** SHAttered (2017). Generation of two valid PDF files with 2^63 operations. + ** Birthday-Near-Collision (2019). This attack allows for chosen prefix + attacks with 2^68 operations. + ** Shambles (2020). This attack allows for chosen prefix attacks with 2^63 + operations. ++ +While we have protections in place against known attacks, it is expected +that more attacks against SHA-1 will be found by future research. Paired +with the ever-growing capability of hardware, it is only a matter of time +before SHA-1 will be considered broken completely. We want to be prepared +and will thus change the default hash algorithm to "sha256" for newly +initialized repositories. ++ +An important requirement for this change is that the ecosystem is ready to +support the "sha256" object format. This includes popular Git libraries, +applications and forges. ++ +There is no plan to deprecate the "sha1" object format at this point in time. ++ +Cf. <2f5de416-04ba-c23d-1e0b-83bb655829a7@zombino.com>, +<20170223155046.e7nxivfwqqoprsqj@LykOS.localdomain>, +<CA+EOSBncr=4a4d8n9xS4FNehyebpmX8JiUwCsXD47EQDE+DiUQ@mail.gmail.com>. + +=== Removals + +* Support for grafting commits has long been superseded by git-replace(1). + Grafts are inferior to replacement refs: ++ + ** Grafts are a local-only mechanism and cannot be shared across + repositories. + ** Grafts can lead to hard-to-diagnose problems when transferring objects + between repositories. ++ +The grafting mechanism has been marked as outdated since e650d0643b (docs: mark +info/grafts as outdated, 2014-03-05) and will be removed. ++ +Cf. <20140304174806.GA11561@sigill.intra.peff.net>. + +== Superseded features that will not be deprecated + +Some features have gained newer replacements that aim to improve the design in +certain ways. The fact that there is a replacement does not automatically mean +that the old way of doing things will eventually be removed. This section tracks +those features with newer alternatives. + +* The features git-checkout(1) offers are covered by the pair of commands + git-restore(1) and git-switch(1). Because the use of git-checkout(1) is still + widespread, and it is not expected that this will change anytime soon, all + three commands will stay. ++ +This decision may get revisited in case we ever figure out that there are +almost no users of any of the commands anymore. ++ +Cf. <xmqqttjazwwa.fsf@gitster.g>, +<xmqqleeubork.fsf@gitster.g>, +<112b6568912a6de6672bf5592c3a718e@manjaro.org>. diff --git a/Documentation/Makefile b/Documentation/Makefile index a3868a462f..dc65759cb1 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -51,6 +51,7 @@ MAN7_TXT += gitdiffcore.txt MAN7_TXT += giteveryday.txt MAN7_TXT += gitfaq.txt MAN7_TXT += gitglossary.txt +MAN7_TXT += gitpacking.txt MAN7_TXT += gitnamespaces.txt MAN7_TXT += gitremote-helpers.txt MAN7_TXT += gitrevisions.txt diff --git a/Documentation/RelNotes/2.46.0.txt b/Documentation/RelNotes/2.46.0.txt index 58fde28f5c..6cccdcd2bb 100644 --- a/Documentation/RelNotes/2.46.0.txt +++ b/Documentation/RelNotes/2.46.0.txt @@ -70,6 +70,25 @@ UI, Workflows & Features files backend for its ref storage to use the reftable backend, with limitations. + * "git diff --exit-code --ext-diff" learned to take the exit status + of the external diff driver into account when deciding the exit + status of the overall "git diff" invocation when configured to do + so. + + * "git update-ref --stdin" learned to handle transactional updates of + symbolic-refs. + + * "git format-patch --interdiff" for multi-patch series learned to + turn on cover letters automatically (unless told never to enable + cover letter with "--no-cover-letter" and such). + + * The "--heads" option of "ls-remote" and "show-ref" has been been + deprecated; "--branches" replaces "--heads". + + * For over a year, setting add.interactive.useBuiltin configuration + variable did nothing but giving a "this does not do anything" + warning. The warning has been removed. + Performance, Internal Implementation, Development Support etc. @@ -149,6 +168,19 @@ Performance, Internal Implementation, Development Support etc. * Building with "-Werror -Wwrite-strings" is now supported. + * To help developers, the build procedure now allows builders to use + CFLAGS_APPEND to specify additional CFLAGS. + + * "oidtree" tests were rewritten to use the unit test framework. + + * The structure of the document that records longer-term project + decisions to deprecate/remove/update various behaviour has been + outlined. + + * The pseudo-merge reachability bitmap to help more efficient storage + of the reachability bitmap in a repository with too many refs has + been added. + Fixes since v2.45 ----------------- @@ -273,6 +305,30 @@ Fixes since v2.45 * A leak in "git imap-send" that somehow escapes LSan has been plugged. + * Setting core.abbrev too early before the repository set-up + (typically in "git clone") caused segfault, which as been + corrected. + + * When the user adds to "git rebase -i" instruction to "pick" a merge + commit, the error experience is not pleasant. Such an error is now + caught earlier in the process that parses the todo list. + (merge 4c063c82e9 pw/rebase-i-error-message later to maint). + + * We forgot to normalize the result of getcwd() to NFC on macOS where + all other paths are normalized, which has been corrected. This still + does not address the case where core.precomposeUnicode configuration + is not defined globally. + (merge 71fa8d2212 tb/precompose-getcwd later to maint). + + * Earlier we stopped using the tree of HEAD as the default source of + attributes in a bare repository, but failed to document it. This + has been corrected. + (merge 5c71d6b63a jc/no-default-attr-tree-in-bare later to maint). + + * "git update-server-info" and "git commit-graph --write" have been + updated to use the tempfile API to avoid leaving cruft after + failing. + * Other code cleanup, docfix, build fix, etc. (merge a5a4cb7b27 rs/diff-parseopts-cleanup later to maint). (merge 55702c543e fa/p4-error later to maint). @@ -283,3 +339,6 @@ Fixes since v2.45 (merge 7150f140f9 mt/t0211-typofix later to maint). (merge d424488901 jc/rev-parse-fatal-doc later to maint). (merge 36d900d2b0 rs/difftool-env-simplify later to maint). + (merge e83055ecb0 ds/doc-add-interactive-singlekey later to maint). + (merge f1160393c1 ds/ahead-behind-fix later to maint). + (merge bf6a86236e jc/worktree-git-path later to maint). diff --git a/Documentation/config.txt b/Documentation/config.txt index 2d2d06d7ee..8c0b3ed807 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -384,6 +384,8 @@ include::config/apply.txt[] include::config/attr.txt[] +include::config/bitmap-pseudo-merge.txt[] + include::config/blame.txt[] include::config/branch.txt[] diff --git a/Documentation/config/add.txt b/Documentation/config/add.txt index e0354ceaed..4d753f006e 100644 --- a/Documentation/config/add.txt +++ b/Documentation/config/add.txt @@ -5,9 +5,3 @@ add.ignore-errors (deprecated):: option of linkgit:git-add[1]. `add.ignore-errors` is deprecated, as it does not follow the usual naming convention for configuration variables. - -add.interactive.useBuiltin:: - Unused configuration variable. Used in Git versions v2.25.0 to - v2.36.0 to enable the built-in version of linkgit:git-add[1]'s - interactive mode, which then became the default in Git - versions v2.37.0 to v2.39.0. diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt index 0e35ae5240..fa61241756 100644 --- a/Documentation/config/advice.txt +++ b/Documentation/config/advice.txt @@ -96,6 +96,8 @@ advice.*:: `pushNonFFCurrent`, `pushNonFFMatching`, `pushAlreadyExists`, `pushFetchFirst`, `pushNeedsForce`, and `pushRefNeedsUpdate` simultaneously. + rebaseTodoError:: + Shown when there is an error after editing the rebase todo list. refSyntax:: Shown when the user provides an illegal ref name, to tell the user about the ref syntax documentation. diff --git a/Documentation/config/attr.txt b/Documentation/config/attr.txt index 1a482d6af2..c4a5857993 100644 --- a/Documentation/config/attr.txt +++ b/Documentation/config/attr.txt @@ -1,7 +1,6 @@ attr.tree:: A reference to a tree in the repository from which to read attributes, - instead of the `.gitattributes` file in the working tree. In a bare - repository, this defaults to `HEAD:.gitattributes`. If the value does - not resolve to a valid tree object, an empty tree is used instead. + instead of the `.gitattributes` file in the working tree. If the value + does not resolve to a valid tree object, an empty tree is used instead. When the `GIT_ATTR_SOURCE` environment variable or `--attr-source` command line option are used, this configuration variable has no effect. diff --git a/Documentation/config/bitmap-pseudo-merge.txt b/Documentation/config/bitmap-pseudo-merge.txt new file mode 100644 index 0000000000..1f264eca99 --- /dev/null +++ b/Documentation/config/bitmap-pseudo-merge.txt @@ -0,0 +1,91 @@ +NOTE: The configuration options in `bitmapPseudoMerge.*` are considered +EXPERIMENTAL and may be subject to change or be removed entirely in the +future. For more information about the pseudo-merge bitmap feature, see +the "Pseudo-merge bitmaps" section of linkgit:gitpacking[7]. + +bitmapPseudoMerge.<name>.pattern:: + Regular expression used to match reference names. Commits + pointed to by references matching this pattern (and meeting + the below criteria, like `bitmapPseudoMerge.<name>.sampleRate` + and `bitmapPseudoMerge.<name>.threshold`) will be considered + for inclusion in a pseudo-merge bitmap. ++ +Commits are grouped into pseudo-merge groups based on whether or not +any reference(s) that point at a given commit match the pattern, which +is an extended regular expression. ++ +Within a pseudo-merge group, commits may be further grouped into +sub-groups based on the capture groups in the pattern. These +sub-groupings are formed from the regular expressions by concatenating +any capture groups from the regular expression, with a '-' dash in +between. ++ +For example, if the pattern is `refs/tags/`, then all tags (provided +they meet the below criteria) will be considered candidates for the +same pseudo-merge group. However, if the pattern is instead +`refs/remotes/([0-9])+/tags/`, then tags from different remotes will +be grouped into separate pseudo-merge groups, based on the remote +number. + +bitmapPseudoMerge.<name>.decay:: + Determines the rate at which consecutive pseudo-merge bitmap + groups decrease in size. Must be non-negative. This parameter + can be thought of as `k` in the function `f(n) = C * n^-k`, + where `f(n)` is the size of the `n`th group. ++ +Setting the decay rate equal to `0` will cause all groups to be the +same size. Setting the decay rate equal to `1` will cause the `n`th +group to be `1/n` the size of the initial group. Higher values of the +decay rate cause consecutive groups to shrink at an increasing rate. +The default is `1`. ++ +If all groups are the same size, it is possible that groups containing +newer commits will be able to be used less often than earlier groups, +since it is more likely that the references pointing at newer commits +will be updated more often than a reference pointing at an old commit. + +bitmapPseudoMerge.<name>.sampleRate:: + Determines the proportion of non-bitmapped commits (among + reference tips) which are selected for inclusion in an + unstable pseudo-merge bitmap. Must be between `0` and `1` + (inclusive). The default is `1`. + +bitmapPseudoMerge.<name>.threshold:: + Determines the minimum age of non-bitmapped commits (among + reference tips, as above) which are candidates for inclusion + in an unstable pseudo-merge bitmap. The default is + `1.week.ago`. + +bitmapPseudoMerge.<name>.maxMerges:: + Determines the maximum number of pseudo-merge commits among + which commits may be distributed. ++ +For pseudo-merge groups whose pattern does not contain any capture +groups, this setting is applied for all commits matching the regular +expression. For patterns that have one or more capture groups, this +setting is applied for each distinct capture group. ++ +For example, if your capture group is `refs/tags/`, then this setting +will distribute all tags into a maximum of `maxMerges` pseudo-merge +commits. However, if your capture group is, say, +`refs/remotes/([0-9]+)/tags/`, then this setting will be applied to +each remote's set of tags individually. ++ +Must be non-negative. The default value is 64. + +bitmapPseudoMerge.<name>.stableThreshold:: + Determines the minimum age of commits (among reference tips, + as above, however stable commits are still considered + candidates even when they have been covered by a bitmap) which + are candidates for a stable a pseudo-merge bitmap. The default + is `1.month.ago`. ++ +Setting this threshold to a smaller value (e.g., 1.week.ago) will cause +more stable groups to be generated (which impose a one-time generation +cost) but those groups will likely become stale over time. Using a +larger value incurs the opposite penalty (fewer stable groups which are +more useful). + +bitmapPseudoMerge.<name>.stableSize:: + Determines the size (in number of commits) of a stable + psuedo-merge bitmap. The default is `512`. diff --git a/Documentation/config/diff.txt b/Documentation/config/diff.txt index 5ce7b91f1d..190bda17e5 100644 --- a/Documentation/config/diff.txt +++ b/Documentation/config/diff.txt @@ -79,6 +79,15 @@ diff.external:: you want to use an external diff program only on a subset of your files, you might want to use linkgit:gitattributes[5] instead. +diff.trustExitCode:: + If this boolean value is set to true then the + `diff.external` command is expected to return exit code + 0 if it considers the input files to be equal or 1 if it + considers them to be different, like `diff(1)`. + If it is set to false, which is the default, then the command + is expected to return exit code 0 regardless of equality. + Any other exit code causes Git to report a fatal error. + diff.ignoreSubmodules:: Sets the default value of --ignore-submodules. Note that this affects only 'git diff' Porcelain, and not lower level 'diff' @@ -164,6 +173,15 @@ diff.<driver>.command:: The custom diff driver command. See linkgit:gitattributes[5] for details. +diff.<driver>.trustExitCode:: + If this boolean value is set to true then the + `diff.<driver>.command` command is expected to return exit code + 0 if it considers the input files to be equal or 1 if it + considers them to be different, like `diff(1)`. + If it is set to false, which is the default, then the command + is expected to return exit code 0 regardless of equality. + Any other exit code causes Git to report a fatal error. + diff.<driver>.xfuncname:: The regular expression that the diff driver should use to recognize the hunk header. A built-in pattern may also be used. diff --git a/Documentation/config/interactive.txt b/Documentation/config/interactive.txt index 5cc26555f1..8b876cb4eb 100644 --- a/Documentation/config/interactive.txt +++ b/Documentation/config/interactive.txt @@ -1,8 +1,8 @@ interactive.singleKey:: - In interactive commands, allow the user to provide one-letter - input with a single key (i.e., without hitting enter). - Currently this is used by the `--patch` mode of - linkgit:git-add[1], linkgit:git-checkout[1], + When set to true, allow the user to provide one-letter input + with a single key (i.e., without hitting the Enter key) in + interactive commands. This is currently used by the `--patch` + mode of linkgit:git-add[1], linkgit:git-checkout[1], linkgit:git-restore[1], linkgit:git-commit[1], linkgit:git-reset[1], and linkgit:git-stash[1]. diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt index c7df20e571..cd0b81adbb 100644 --- a/Documentation/diff-options.txt +++ b/Documentation/diff-options.txt @@ -820,6 +820,11 @@ ifndef::git-log[] --quiet:: Disable all output of the program. Implies `--exit-code`. + Disables execution of external diff helpers whose exit code + is not trusted, i.e. their respective configuration option + `diff.trustExitCode` or `diff.<driver>.trustExitCode` or + environment variable `GIT_EXTERNAL_DIFF_TRUST_EXIT_CODE` is + false. endif::git-log[] endif::git-format-patch[] diff --git a/Documentation/git-ls-remote.txt b/Documentation/git-ls-remote.txt index 1c4f696ab5..76c86c3ce4 100644 --- a/Documentation/git-ls-remote.txt +++ b/Documentation/git-ls-remote.txt @@ -9,7 +9,7 @@ git-ls-remote - List references in a remote repository SYNOPSIS -------- [verse] -'git ls-remote' [--heads] [--tags] [--refs] [--upload-pack=<exec>] +'git ls-remote' [--branches] [--tags] [--refs] [--upload-pack=<exec>] [-q | --quiet] [--exit-code] [--get-url] [--sort=<key>] [--symref] [<repository> [<patterns>...]] @@ -21,14 +21,16 @@ commit IDs. OPTIONS ------- --h:: ---heads:: +-b:: +--branches:: -t:: --tags:: - Limit to only refs/heads and refs/tags, respectively. + Limit to only local branches and local tags, respectively. These options are _not_ mutually exclusive; when given both, references stored in refs/heads and refs/tags are - displayed. Note that `git ls-remote -h` used without + displayed. Note that `--heads` and `-h` are deprecated + synonyms for `--branches` and `-b` and may be removed in + the future. Also note that `git ls-remote -h` used without anything else on the command line gives help, consistent with other git subcommands. diff --git a/Documentation/git-show-ref.txt b/Documentation/git-show-ref.txt index ba75747005..616d919655 100644 --- a/Documentation/git-show-ref.txt +++ b/Documentation/git-show-ref.txt @@ -9,8 +9,8 @@ SYNOPSIS -------- [verse] 'git show-ref' [--head] [-d | --dereference] - [-s | --hash[=<n>]] [--abbrev[=<n>]] [--tags] - [--heads] [--] [<pattern>...] + [-s | --hash[=<n>]] [--abbrev[=<n>]] [--branches] [--tags] + [--] [<pattern>...] 'git show-ref' --verify [-q | --quiet] [-d | --dereference] [-s | --hash[=<n>]] [--abbrev[=<n>]] [--] [<ref>...] @@ -45,12 +45,14 @@ OPTIONS Show the HEAD reference, even if it would normally be filtered out. ---heads:: +--branches:: --tags:: - Limit to "refs/heads" and "refs/tags", respectively. These options + Limit to local branches and local tags, respectively. These options are not mutually exclusive; when given both, references stored in - "refs/heads" and "refs/tags" are displayed. + "refs/heads" and "refs/tags" are displayed. Note that `--heads` + is a deprecated synonym for `--branches` and may be removed + in the future. -d:: --dereference:: @@ -139,7 +141,7 @@ When using `--hash` (and not `--dereference`), the output is in the format: For example, ----------------------------------------------------------------------------- -$ git show-ref --heads --hash +$ git show-ref --branches --hash 2e3ba0114a1f52b47df29743d6915d056be13278 185008ae97960c8d551adcd9e23565194651b5d1 03adf42c988195b50e1a1935ba5fcbc39b2b029b @@ -183,8 +185,8 @@ to check whether a particular branch exists or not (notice how we don't actually want to show any results, and we want to use the full refname for it in order to not trigger the problem with ambiguous partial matches). -To show only tags, or only proper branch heads, use `--tags` and/or `--heads` -respectively (using both means that it shows tags and heads, but not other +To show only tags, or only proper branch heads, use `--tags` and/or `--branches` +respectively (using both means that it shows tags and branches, but not other random references under the refs/ subdirectory). To do automatic tag object dereferencing, use the `-d` or `--dereference` diff --git a/Documentation/git-update-ref.txt b/Documentation/git-update-ref.txt index 374a2ebd2b..afcf33cf60 100644 --- a/Documentation/git-update-ref.txt +++ b/Documentation/git-update-ref.txt @@ -65,6 +65,10 @@ performs all modifications together. Specify commands of the form: create SP <ref> SP <new-oid> LF delete SP <ref> [SP <old-oid>] LF verify SP <ref> [SP <old-oid>] LF + symref-update SP <ref> SP <new-target> [SP (ref SP <old-target> | oid SP <old-oid>)] LF + symref-create SP <ref> SP <new-target> LF + symref-delete SP <ref> [SP <old-target>] LF + symref-verify SP <ref> [SP <old-target>] LF option SP <opt> LF start LF prepare LF @@ -86,6 +90,10 @@ quoting: create SP <ref> NUL <new-oid> NUL delete SP <ref> NUL [<old-oid>] NUL verify SP <ref> NUL [<old-oid>] NUL + symref-update SP <ref> NUL <new-target> [NUL (ref NUL <old-target> | oid NUL <old-oid>)] NUL + symref-create SP <ref> NUL <new-target> NUL + symref-delete SP <ref> [NUL <old-target>] NUL + symref-verify SP <ref> [NUL <old-target>] NUL option SP <opt> NUL start NUL prepare NUL @@ -113,10 +121,27 @@ delete:: Delete <ref> after verifying it exists with <old-oid>, if given. If given, <old-oid> may not be zero. +symref-update:: + Set <ref> to <new-target> after verifying <old-target> or <old-oid>, + if given. Specify a zero <old-oid> to ensure that the ref does not + exist before the update. + verify:: Verify <ref> against <old-oid> but do not change it. If <old-oid> is zero or missing, the ref must not exist. +symref-create: + Create symbolic ref <ref> with <new-target> after verifying + it does not exist. + +symref-delete:: + Delete <ref> after verifying it exists with <old-target>, if given. + +symref-verify:: + Verify symbolic <ref> against <old-target> but do not change it. + If <old-target> is missing, the ref must not exist. Can only be + used in `no-deref` mode. + option:: Modify the behavior of the next command naming a <ref>. The only valid option is `no-deref` to avoid dereferencing diff --git a/Documentation/git.txt b/Documentation/git.txt index a31a70acca..4489e2297a 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -644,6 +644,16 @@ parameter, <path>. For each path `GIT_EXTERNAL_DIFF` is called, two environment variables, `GIT_DIFF_PATH_COUNTER` and `GIT_DIFF_PATH_TOTAL` are set. +`GIT_EXTERNAL_DIFF_TRUST_EXIT_CODE`:: + If this Boolean environment variable is set to true then the + `GIT_EXTERNAL_DIFF` command is expected to return exit code + 0 if it considers the input files to be equal or 1 if it + considers them to be different, like `diff(1)`. + If it is set to false, which is the default, then the command + is expected to return exit code 0 regardless of equality. + Any other exit code causes Git to report a fatal error. + + `GIT_DIFF_PATH_COUNTER`:: A 1-based counter incremented by one for every path. diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt index 4338d023d9..80cae17f37 100644 --- a/Documentation/gitattributes.txt +++ b/Documentation/gitattributes.txt @@ -776,6 +776,11 @@ with the above configuration, i.e. `j-c-diff`, with 7 parameters, just like `GIT_EXTERNAL_DIFF` program is called. See linkgit:git[1] for details. +If the program is able to ignore certain changes (similar to +`git diff --ignore-space-change`), then also set the option +`trustExitCode` to true. It is then expected to return exit code 1 if +it finds significant changes and 0 if it doesn't. + Setting the internal diff algorithm ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/Documentation/gitpacking.txt b/Documentation/gitpacking.txt new file mode 100644 index 0000000000..4a6fcba6f7 --- /dev/null +++ b/Documentation/gitpacking.txt @@ -0,0 +1,189 @@ +gitpacking(7) +============= + +NAME +---- +gitpacking - Advanced concepts related to packing in Git + +SYNOPSIS +-------- +gitpacking + +DESCRIPTION +----------- + +This document aims to describe some advanced concepts related to packing +in Git. + +Many concepts are currently described scattered between manual pages of +various Git commands, including linkgit:git-pack-objects[1], +linkgit:git-repack[1], and others, as well as linkgit:gitformat-pack[5], +and parts of the `Documentation/technical` tree. + +There are many aspects of packing in Git that are not covered in this +document that instead live in the aforementioned areas. Over time, those +scattered bits may coalesce into this document. + +== Pseudo-merge bitmaps + +NOTE: Pseudo-merge bitmaps are considered an experimental feature, so +the configuration and many of the ideas are subject to change. + +=== Background + +Reachability bitmaps are most efficient when we have on-disk stored +bitmaps for one or more of the starting points of a traversal. For this +reason, Git prefers storing bitmaps for commits at the tips of refs, +because traversals tend to start with those points. + +But if you have a large number of refs, it's not feasible to store a +bitmap for _every_ ref tip. It takes up space, and just OR-ing all of +those bitmaps together is expensive. + +One way we can deal with that is to create bitmaps that represent +_groups_ of refs. When a traversal asks about the entire group, then we +can use this single bitmap instead of considering each ref individually. +Because these bitmaps represent the set of objects which would be +reachable in a hypothetical merge of all of the commits, we call them +pseudo-merge bitmaps. + +=== Overview + +A "pseudo-merge bitmap" is used to refer to a pair of bitmaps, as +follows: + +Commit bitmap:: + + A bitmap whose set bits describe the set of commits included in the + pseudo-merge's "merge" bitmap (as below). + +Merge bitmap:: + + A bitmap whose set bits describe the reachability closure over the set + of commits in the pseudo-merge's "commits" bitmap (as above). An + identical bitmap would be generated for an octopus merge with the same + set of parents as described in the commits bitmap. + +Pseudo-merge bitmaps can accelerate bitmap traversals when all commits +for a given pseudo-merge are listed on either side of the traversal, +either directly (by explicitly asking for them as part of the `HAVES` +or `WANTS`) or indirectly (by encountering them during a fill-in +traversal). + +=== Use-cases + +For example, suppose there exists a pseudo-merge bitmap with a large +number of commits, all of which are listed in the `WANTS` section of +some bitmap traversal query. When pseudo-merge bitmaps are enabled, the +bitmap machinery can quickly determine there is a pseudo-merge which +satisfies some subset of the wanted objects on either side of the query. +Then, we can inflate the EWAH-compressed bitmap, and `OR` it in to the +resulting bitmap. By contrast, without pseudo-merge bitmaps, we would +have to repeat the decompression and `OR`-ing step over a potentially +large number of individual bitmaps, which can take proportionally more +time. + +Another benefit of pseudo-merges arises when there is some combination +of (a) a large number of references, with (b) poor bitmap coverage, and +(c) deep, nested trees, making fill-in traversal relatively expensive. +For example, suppose that there are a large enough number of tags where +bitmapping each of the tags individually is infeasible. Without +pseudo-merge bitmaps, computing the result of, say, `git rev-list +--use-bitmap-index --count --objects --tags` would likely require a +large amount of fill-in traversal. But when a large quantity of those +tags are stored together in a pseudo-merge bitmap, the bitmap machinery +can take advantage of the fact that we only care about the union of +objects reachable from all of those tags, and answer the query much +faster. + +=== Configuration + +Reference tips are grouped into different pseudo-merge groups according +to two criteria. A reference name matches one or more of the defined +pseudo-merge patterns, and optionally one or more capture groups within +that pattern which further partition the group. + +Within a group, commits may be considered "stable", or "unstable" +depending on their age. These are adjusted by setting the +`bitmapPseudoMerge.<name>.stableThreshold` and +`bitmapPseudoMerge.<name>.threshold` configuration values, respectively. + +All stable commits are grouped into pseudo-merges of equal size +(`bitmapPseudoMerge.<name>.stableSize`). If the `stableSize` +configuration is set to, say, 100, then the first 100 commits (ordered +by committer date) which are older than the `stableThreshold` value will +form one group, the next 100 commits will form another group, and so on. + +Among unstable commits, the pseudo-merge machinery will attempt to +combine older commits into large groups as opposed to newer commits +which will appear in smaller groups. This is based on the heuristic that +references whose tip commit is older are less likely to be modified to +point at a different commit than a reference whose tip commit is newer. + +The size of groups is determined by a power-law decay function, and the +decay parameter roughly corresponds to "k" in `f(n) = C*n^(-k/100)`, +where `f(n)` describes the size of the `n`-th pseudo-merge group. The +sample rate controls what percentage of eligible commits are considered +as candidates. The threshold parameter indicates the minimum age (so as +to avoid including too-recent commits in a pseudo-merge group, making it +less likely to be valid). The "maxMerges" parameter sets an upper-bound +on the number of pseudo-merge commits an individual group + +The "stable"-related parameters control "stable" pseudo-merge groups, +comprised of a fixed number of commits which are older than the +configured "stable threshold" value and may be grouped together in +chunks of "stableSize" in order of age. + +The exact configuration for pseudo-merges is as follows: + +include::config/bitmap-pseudo-merge.txt[] + +=== Examples + +Suppose that you have a repository with a large number of references, +and you want a bare-bones configuration of pseudo-merge bitmaps that +will enhance bitmap coverage of the `refs/` namespace. You may start +wiht a configuration like so: + + [bitmapPseudoMerge "all"] + pattern = "refs/" + threshold = now + stableThreshold = never + sampleRate = 100 + maxMerges = 64 + +This will create pseudo-merge bitmaps for all references, regardless of +their age, and group them into 64 pseudo-merge commits. + +If you wanted to separate tags from branches when generating +pseudo-merge commits, you would instead define the pattern with a +capture group, like so: + + [bitmapPseudoMerge "all"] + pattern = "refs/(heads/tags)/" + +Suppose instead that you are working in a fork-network repository, with +each fork specified by some numeric ID, and whose refs reside in +`refs/virtual/NNN/` (where `NNN` is the numeric ID corresponding to some +fork) in the network. In this instance, you may instead write something +like: + + [bitmapPseudoMerge "all"] + pattern = "refs/virtual/([0-9]+)/(heads|tags)/" + threshold = now + stableThreshold = never + sampleRate = 100 + maxMerges = 64 + +Which would generate pseudo-merge group identifiers like "1234-heads", +and "5678-tags" (for branches in fork "1234", and tags in remote "5678", +respectively). + +SEE ALSO +-------- +linkgit:git-pack-objects[1] +linkgit:git-repack[1] + +GIT +--- +Part of the linkgit:git[1] suite diff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt index f5d200939b..bfb0ec7beb 100644 --- a/Documentation/technical/bitmap-format.txt +++ b/Documentation/technical/bitmap-format.txt @@ -255,3 +255,144 @@ triplet is - xor_row (4 byte integer, network byte order): :: The position of the triplet whose bitmap is used to compress this one, or `0xffffffff` if no such bitmap exists. + +Pseudo-merge bitmaps +-------------------- + +If the `BITMAP_OPT_PSEUDO_MERGES` flag is set, a variable number of +bytes (preceding the name-hash cache, commit lookup table, and trailing +checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps. + +For more information on what pseudo-merges are, why they are useful, and +how to configure them, see the information in linkgit:gitpacking[7]. + +=== File format + +If enabled, pseudo-merge bitmaps are stored in an optional section at +the end of a `.bitmap` file. The format is as follows: + +.... ++-------------------------------------------+ +| .bitmap File | ++-------------------------------------------+ +| | +| Pseudo-merge bitmaps (Variable Length) | +| +---------------------------+ | +| | commits_bitmap (EWAH) | | +| +---------------------------+ | +| | merge_bitmap (EWAH) | | +| +---------------------------+ | +| | ++-------------------------------------------+ +| | +| Lookup Table | +| +---------------------------+ | +| | commit_pos (4 bytes) | | +| +---------------------------+ | +| | offset (8 bytes) | | +| +------------+--------------+ | +| | +| Offset Cases: | +| ------------- | +| | +| 1. MSB Unset: single pseudo-merge bitmap | +| + offset to pseudo-merge bitmap | +| | +| 2. MSB Set: multiple pseudo-merges | +| + offset to extended lookup table | +| | ++-------------------------------------------+ +| | +| Extended Lookup Table (Optional) | +| +----+----------+----------+----------+ | +| | N | Offset 1 | .... | Offset N | | +| +----+----------+----------+----------+ | +| | | 8 bytes | .... | 8 bytes | | +| +----+----------+----------+----------+ | +| | ++-------------------------------------------+ +| | +| Pseudo-merge position table | +| +----+----------+----------+----------+ | +| | N | Offset 1 | .... | Offset N | | +| +----+----------+----------+----------+ | +| | | 8 bytes | .... | 8 bytes | | +| +----+----------+----------+----------+ | +| | ++-------------------------------------------+ +| | +| Pseudo-merge Metadata | +| +-----------------------------------+ | +| | # pseudo-merges (4 bytes) | | +| +-----------------------------------+ | +| | # commits (4 bytes) | | +| +-----------------------------------+ | +| | Lookup offset (8 bytes) | | +| +-----------------------------------+ | +| | Extension size (8 bytes) | | +| +-----------------------------------+ | +| | ++-------------------------------------------+ +.... + +* One or more pseudo-merge bitmaps, each containing: + + ** `commits_bitmap`, an EWAH-compressed bitmap describing the set of + commits included in the this psuedo-merge. + + ** `merge_bitmap`, an EWAH-compressed bitmap describing the union of + the set of objects reachable from all commits listed in the + `commits_bitmap`. + +* A lookup table, mapping pseudo-merged commits to the pseudo-merges + they belong to. Entries appear in increasing order of each commit's + bit position. Each entry is 12 bytes wide, and is comprised of the + following: + + ** `commit_pos`, a 4-byte unsigned value (in network byte-order) + containing the bit position for this commit. + + ** `offset`, an 8-byte unsigned value (also in network byte-order) + containing either one of two possible offsets, depending on whether or + not the most-significant bit is set. + + *** If unset (i.e. `offset & ((uint64_t)1<<63) == 0`), the offset + (relative to the beginning of the `.bitmap` file) at which the + pseudo-merge bitmap for this commit can be read. This indicates + only a single pseudo-merge bitmap contains this commit. + + *** If set (i.e. `offset & ((uint64_t)1<<63) != 0`), the offset + (again relative to the beginning of the `.bitmap` file) at which + the extended offset table can be located describing the set of + pseudo-merge bitmaps which contain this commit. This indicates + that multiple pseudo-merge bitmaps contain this commit. + +* An (optional) extended lookup table (written if and only if there is + at least one commit which appears in more than one pseudo-merge). + There are as many entries as commits which appear in multiple + pseudo-merges. Each entry contains the following: + + ** `N`, a 4-byte unsigned value equal to the number of pseudo-merges + which contain a given commit. + + ** An array of `N` 8-byte unsigned values, each of which is + interpreted as an offset (relative to the beginning of the + `.bitmap` file) at which a pseudo-merge bitmap for this commit can + be read. These values occur in no particular order. + +* Positions for all pseudo-merges, each stored as an 8-byte unsigned + value (in network byte-order) containing the offset (relative to the + beginning of the `.bitmap` file) of each consecutive pseudo-merge. + +* A 4-byte unsigned value (in network byte-order) equal to the number of + pseudo-merges. + +* A 4-byte unsigned value (in network byte-order) equal to the number of + unique commits which appear in any pseudo-merge. + +* An 8-byte unsigned value (in network byte-order) equal to the number + of bytes between the start of the pseudo-merge section and the + beginning of the lookup table. + +* An 8-byte unsigned value (in network byte-order) equal to the number + of bytes in the pseudo-merge section (including this field). @@ -810,7 +810,6 @@ TEST_BUILTINS_OBJS += test-mergesort.o TEST_BUILTINS_OBJS += test-mktemp.o TEST_BUILTINS_OBJS += test-oid-array.o TEST_BUILTINS_OBJS += test-oidmap.o -TEST_BUILTINS_OBJS += test-oidtree.o TEST_BUILTINS_OBJS += test-online-cpus.o TEST_BUILTINS_OBJS += test-pack-mtimes.o TEST_BUILTINS_OBJS += test-parse-options.o @@ -1104,6 +1103,7 @@ LIB_OBJS += prompt.o LIB_OBJS += protocol.o LIB_OBJS += protocol-caps.o LIB_OBJS += prune-packed.o +LIB_OBJS += pseudo-merge.o LIB_OBJS += quote.o LIB_OBJS += range-diff.o LIB_OBJS += reachable.o @@ -1337,6 +1337,7 @@ UNIT_TEST_PROGRAMS += t-ctype UNIT_TEST_PROGRAMS += t-example-decorate UNIT_TEST_PROGRAMS += t-hash UNIT_TEST_PROGRAMS += t-mem-pool +UNIT_TEST_PROGRAMS += t-oidtree UNIT_TEST_PROGRAMS += t-prio-queue UNIT_TEST_PROGRAMS += t-reftable-basics UNIT_TEST_PROGRAMS += t-strbuf @@ -1346,6 +1347,7 @@ UNIT_TEST_PROGRAMS += t-trailer UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_BIN)/%$X,$(UNIT_TEST_PROGRAMS)) UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS)) UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/test-lib.o +UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/lib-oid.o # xdiff and reftable libs may in turn depend on what is in libgit.a GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(LIB_FILE) @@ -1449,8 +1451,8 @@ ALL_COMMANDS_TO_INSTALL += git-upload-archive$(X) ALL_COMMANDS_TO_INSTALL += git-upload-pack$(X) endif -ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS) -ALL_LDFLAGS = $(LDFLAGS) +ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS) $(CFLAGS_APPEND) +ALL_LDFLAGS = $(LDFLAGS) $(LDFLAGS_APPEND) ifdef SANITIZE SANITIZERS := $(foreach flag,$(subst $(comma),$(space),$(SANITIZE)),$(flag)) @@ -3849,7 +3851,10 @@ $(FUZZ_PROGRAMS): %: %.o oss-fuzz/dummy-cmd-main.o $(GITLIBS) GIT-LDFLAGS -Wl,--allow-multiple-definition \ $(filter %.o,$^) $(filter %.a,$^) $(LIBS) $(LIB_FUZZING_ENGINE) -$(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-LDFLAGS +$(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o \ + $(UNIT_TEST_DIR)/test-lib.o \ + $(UNIT_TEST_DIR)/lib-oid.o \ + $(GITLIBS) GIT-LDFLAGS $(call mkdir_p_parent_template) $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \ $(filter %.o,$^) $(filter %.a,$^) $(LIBS) @@ -70,6 +70,7 @@ static struct { [ADVICE_PUSH_UNQUALIFIED_REF_NAME] = { "pushUnqualifiedRefName" }, [ADVICE_PUSH_UPDATE_REJECTED] = { "pushUpdateRejected" }, [ADVICE_PUSH_UPDATE_REJECTED_ALIAS] = { "pushNonFastForward" }, /* backwards compatibility */ + [ADVICE_REBASE_TODO_ERROR] = { "rebaseTodoError" }, [ADVICE_REF_SYNTAX] = { "refSyntax" }, [ADVICE_RESET_NO_REFRESH_WARNING] = { "resetNoRefresh" }, [ADVICE_RESOLVE_CONFLICT] = { "resolveConflict" }, @@ -37,6 +37,7 @@ enum advice_type { ADVICE_PUSH_UNQUALIFIED_REF_NAME, ADVICE_PUSH_UPDATE_REJECTED, ADVICE_PUSH_UPDATE_REJECTED_ALIAS, + ADVICE_REBASE_TODO_ERROR, ADVICE_REF_SYNTAX, ADVICE_RESET_NO_REFRESH_WARNING, ADVICE_RESOLVE_CONFLICT, @@ -865,7 +865,8 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate, stack = read_attr_from_blob(istate, &istate->cache[sparse_dir_pos]->oid, relative_path, flags); } else { buf = read_blob_data_from_index(istate, path, &size); - stack = read_attr_from_buf(buf, size, path, flags); + if (buf) + stack = read_attr_from_buf(buf, size, path, flags); } return stack; } diff --git a/builtin/add.c b/builtin/add.c index 3dfcfc5fba..40b61ef90d 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -150,11 +150,7 @@ static int refresh(int verbose, const struct pathspec *pathspec) int interactive_add(const char **argv, const char *prefix, int patch) { struct pathspec pathspec; - int unused, ret; - - if (!git_config_get_bool("add.interactive.usebuiltin", &unused)) - warning(_("the add.interactive.useBuiltin setting has been removed!\n" - "See its entry in 'git help config' for details.")); + int ret; parse_pathspec(&pathspec, 0, PATHSPEC_PREFER_FULL | diff --git a/builtin/clone.c b/builtin/clone.c index 45d78614a4..b28f88eb43 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -580,7 +580,7 @@ static void write_remote_refs(const struct ref *local_refs) if (!r->peer_ref) continue; if (ref_transaction_create(t, r->peer_ref->name, &r->old_oid, - 0, NULL, &err)) + NULL, 0, NULL, &err)) die("%s", err.buf); } diff --git a/builtin/fetch.c b/builtin/fetch.c index 0c061b2ed0..693f02b958 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1391,8 +1391,8 @@ static int prune_refs(struct display_state *display_state, if (!dry_run) { if (transaction) { for (ref = stale_refs; ref; ref = ref->next) { - result = ref_transaction_delete(transaction, ref->name, NULL, 0, - "fetch: prune", &err); + result = ref_transaction_delete(transaction, ref->name, NULL, + NULL, 0, "fetch: prune", &err); if (result) goto cleanup; } diff --git a/builtin/log.c b/builtin/log.c index b8846a9458..21a81c72ac 100644 --- a/builtin/log.c +++ b/builtin/log.c @@ -2382,6 +2382,8 @@ int cmd_format_patch(int argc, const char **argv, const char *prefix) if (cover_letter == -1) { if (cfg.config_cover_letter == COVER_AUTO) cover_letter = (total > 1); + else if ((idiff_prev.nr || rdiff_prev) && (total > 1)) + cover_letter = (cfg.config_cover_letter != COVER_OFF); else cover_letter = (cfg.config_cover_letter == COVER_ON); } diff --git a/builtin/ls-remote.c b/builtin/ls-remote.c index e8d65ebbdc..69841ed49f 100644 --- a/builtin/ls-remote.c +++ b/builtin/ls-remote.c @@ -9,7 +9,7 @@ #include "wildmatch.h" static const char * const ls_remote_usage[] = { - N_("git ls-remote [--heads] [--tags] [--refs] [--upload-pack=<exec>]\n" + N_("git ls-remote [--branches] [--tags] [--refs] [--upload-pack=<exec>]\n" " [-q | --quiet] [--exit-code] [--get-url] [--sort=<key>]\n" " [--symref] [<repository> [<patterns>...]]"), NULL @@ -68,7 +68,10 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix) N_("path of git-upload-pack on the remote host"), PARSE_OPT_HIDDEN }, OPT_BIT('t', "tags", &flags, N_("limit to tags"), REF_TAGS), - OPT_BIT('h', "heads", &flags, N_("limit to heads"), REF_HEADS), + OPT_BIT('b', "branches", &flags, N_("limit to branches"), REF_BRANCHES), + OPT_BIT_F('h', "heads", &flags, + N_("deprecated synonym for --branches"), REF_BRANCHES, + PARSE_OPT_HIDDEN), OPT_BIT(0, "refs", &flags, N_("do not show peeled tags"), REF_NORMAL), OPT_BOOL(0, "get-url", &get_url, N_("take url.<base>.insteadOf into account")), @@ -100,7 +103,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix) if (flags & REF_TAGS) strvec_push(&transport_options.ref_prefixes, "refs/tags/"); - if (flags & REF_HEADS) + if (flags & REF_BRANCHES) strvec_push(&transport_options.ref_prefixes, "refs/heads/"); remote = remote_get(dest); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 638f5c57f0..dd97fdd5d2 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1341,7 +1341,8 @@ static void write_pack_file(void) hash_to_hex(hash)); if (write_bitmap_index) { - bitmap_writer_init(&bitmap_writer); + bitmap_writer_init(&bitmap_writer, + the_repository); bitmap_writer_set_checksum(&bitmap_writer, hash); bitmap_writer_build_type_index(&bitmap_writer, &to_pack, written_list, nr_written); diff --git a/builtin/rebase.c b/builtin/rebase.c index 4506bae768..e3a8e74cfc 100644 --- a/builtin/rebase.c +++ b/builtin/rebase.c @@ -206,7 +206,7 @@ static struct replay_opts get_replay_opts(const struct rebase_options *opts) return replay; } -static int edit_todo_file(unsigned flags) +static int edit_todo_file(unsigned flags, struct replay_opts *opts) { const char *todo_file = rebase_path_todo(); struct todo_list todo_list = TODO_LIST_INIT, @@ -217,7 +217,8 @@ static int edit_todo_file(unsigned flags) return error_errno(_("could not read '%s'."), todo_file); strbuf_stripspace(&todo_list.buf, comment_line_str); - res = edit_todo_list(the_repository, &todo_list, &new_todo, NULL, NULL, flags); + res = edit_todo_list(the_repository, opts, &todo_list, &new_todo, + NULL, NULL, flags); if (!res && todo_list_write_to_file(the_repository, &new_todo, todo_file, NULL, NULL, -1, flags & ~(TODO_LIST_SHORTEN_IDS))) res = error_errno(_("could not write '%s'"), todo_file); @@ -308,8 +309,8 @@ static int do_interactive_rebase(struct rebase_options *opts, unsigned flags) error(_("could not generate todo list")); else { discard_index(the_repository->index); - if (todo_list_parse_insn_buffer(the_repository, todo_list.buf.buf, - &todo_list)) + if (todo_list_parse_insn_buffer(the_repository, &replay, + todo_list.buf.buf, &todo_list)) BUG("unusable todo list"); ret = complete_action(the_repository, &replay, flags, @@ -364,9 +365,13 @@ static int run_sequencer_rebase(struct rebase_options *opts) replay_opts_release(&replay_opts); break; } - case ACTION_EDIT_TODO: - ret = edit_todo_file(flags); + case ACTION_EDIT_TODO: { + struct replay_opts replay_opts = get_replay_opts(opts); + + ret = edit_todo_file(flags, &replay_opts); + replay_opts_release(&replay_opts); break; + } case ACTION_SHOW_CURRENT_PATCH: { struct child_process cmd = CHILD_PROCESS_INIT; diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index c8d12ee0a7..f67f37febd 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -1576,7 +1576,8 @@ static const char *update(struct command *cmd, struct shallow_info *si) if (ref_transaction_delete(transaction, namespaced_name, old_oid, - 0, "push", &err)) { + NULL, 0, + "push", &err)) { rp_error("%s", err.buf); ret = "failed to delete"; } else { diff --git a/builtin/show-ref.c b/builtin/show-ref.c index 3114bdc391..839a5c29f3 100644 --- a/builtin/show-ref.c +++ b/builtin/show-ref.c @@ -11,8 +11,8 @@ static const char * const show_ref_usage[] = { N_("git show-ref [--head] [-d | --dereference]\n" - " [-s | --hash[=<n>]] [--abbrev[=<n>]] [--tags]\n" - " [--heads] [--] [<pattern>...]"), + " [-s | --hash[=<n>]] [--abbrev[=<n>]] [--branches] [--tags]\n" + " [--] [<pattern>...]"), N_("git show-ref --verify [-q | --quiet] [-d | --dereference]\n" " [-s | --hash[=<n>]] [--abbrev[=<n>]]\n" " [--] [<ref>...]"), @@ -189,7 +189,7 @@ static int cmd_show_ref__verify(const struct show_one_options *show_one_opts, struct patterns_options { int show_head; - int heads_only; + int branches_only; int tags_only; }; @@ -208,8 +208,8 @@ static int cmd_show_ref__patterns(const struct patterns_options *opts, if (opts->show_head) refs_head_ref(get_main_ref_store(the_repository), show_ref, &show_ref_data); - if (opts->heads_only || opts->tags_only) { - if (opts->heads_only) + if (opts->branches_only || opts->tags_only) { + if (opts->branches_only) refs_for_each_fullref_in(get_main_ref_store(the_repository), "refs/heads/", NULL, show_ref, &show_ref_data); @@ -293,8 +293,10 @@ int cmd_show_ref(int argc, const char **argv, const char *prefix) struct show_one_options show_one_opts = {0}; int verify = 0, exists = 0; const struct option show_ref_options[] = { - OPT_BOOL(0, "tags", &patterns_opts.tags_only, N_("only show tags (can be combined with heads)")), - OPT_BOOL(0, "heads", &patterns_opts.heads_only, N_("only show heads (can be combined with tags)")), + OPT_BOOL(0, "tags", &patterns_opts.tags_only, N_("only show tags (can be combined with branches)")), + OPT_BOOL(0, "branches", &patterns_opts.branches_only, N_("only show branches (can be combined with tags)")), + OPT_HIDDEN_BOOL(0, "heads", &patterns_opts.branches_only, + N_("deprecated synonym for --branches")), OPT_BOOL(0, "exists", &exists, N_("check for reference existence without resolving")), OPT_BOOL(0, "verify", &verify, N_("stricter reference checking, " "requires exact ref path")), diff --git a/builtin/update-ref.c b/builtin/update-ref.c index 6cda1c08aa..471fa5c8d1 100644 --- a/builtin/update-ref.c +++ b/builtin/update-ref.c @@ -77,6 +77,65 @@ static char *parse_refname(const char **next) } /* + * Wrapper around parse_refname which skips the next delimiter. + */ +static char *parse_next_refname(const char **next) +{ + if (line_termination) { + /* Without -z, consume SP and use next argument */ + if (!**next || **next == line_termination) + return NULL; + if (**next != ' ') + die("expected SP but got: %s", *next); + } else { + /* With -z, read the next NUL-terminated line */ + if (**next) + return NULL; + } + /* Skip the delimiter */ + (*next)++; + + return parse_refname(next); +} + +/* + * Wrapper around parse_arg which skips the next delimiter. + */ +static char *parse_next_arg(const char **next) +{ + struct strbuf arg = STRBUF_INIT; + + if (line_termination) { + /* Without -z, consume SP and use next argument */ + if (!**next || **next == line_termination) + return NULL; + if (**next != ' ') + die("expected SP but got: %s", *next); + } else { + /* With -z, read the next NUL-terminated line */ + if (**next) + return NULL; + } + /* Skip the delimiter */ + (*next)++; + + if (line_termination) { + /* Without -z, use the next argument */ + *next = parse_arg(*next, &arg); + } else { + /* With -z, use everything up to the next NUL */ + strbuf_addstr(&arg, *next); + *next += arg.len; + } + + if (arg.len) + return strbuf_detach(&arg, NULL); + + strbuf_release(&arg); + return NULL; +} + +/* * The value being parsed is <old-oid> (as opposed to <new-oid>; the * difference affects which error messages are generated): */ @@ -214,6 +273,61 @@ static void parse_cmd_update(struct ref_transaction *transaction, strbuf_release(&err); } +static void parse_cmd_symref_update(struct ref_transaction *transaction, + const char *next, const char *end) +{ + char *refname, *new_target, *old_arg; + char *old_target = NULL; + struct strbuf err = STRBUF_INIT; + struct object_id old_oid; + int have_old_oid = 0; + + refname = parse_refname(&next); + if (!refname) + die("symref-update: missing <ref>"); + + new_target = parse_next_refname(&next); + if (!new_target) + die("symref-update %s: missing <new-target>", refname); + + old_arg = parse_next_arg(&next); + if (old_arg) { + old_target = parse_next_arg(&next); + if (!old_target) + die("symref-update %s: expected old value", refname); + + if (!strcmp(old_arg, "oid")) { + if (repo_get_oid(the_repository, old_target, &old_oid)) + die("symref-update %s: invalid oid: %s", refname, old_target); + + have_old_oid = 1; + } else if (!strcmp(old_arg, "ref")) { + if (check_refname_format(old_target, REFNAME_ALLOW_ONELEVEL)) + die("symref-update %s: invalid ref: %s", refname, old_target); + } else { + die("symref-update %s: invalid arg '%s' for old value", refname, old_arg); + } + } + + if (*next != line_termination) + die("symref-update %s: extra input: %s", refname, next); + + if (ref_transaction_update(transaction, refname, NULL, + have_old_oid ? &old_oid : NULL, + new_target, + have_old_oid ? NULL : old_target, + update_flags | create_reflog_flag, + msg, &err)) + die("%s", err.buf); + + update_flags = default_flags; + free(refname); + free(old_arg); + free(old_target); + free(new_target); + strbuf_release(&err); +} + static void parse_cmd_create(struct ref_transaction *transaction, const char *next, const char *end) { @@ -234,13 +348,42 @@ static void parse_cmd_create(struct ref_transaction *transaction, if (*next != line_termination) die("create %s: extra input: %s", refname, next); - if (ref_transaction_create(transaction, refname, &new_oid, + if (ref_transaction_create(transaction, refname, &new_oid, NULL, + update_flags | create_reflog_flag, + msg, &err)) + die("%s", err.buf); + + update_flags = default_flags; + free(refname); + strbuf_release(&err); +} + + +static void parse_cmd_symref_create(struct ref_transaction *transaction, + const char *next, const char *end) +{ + struct strbuf err = STRBUF_INIT; + char *refname, *new_target; + + refname = parse_refname(&next); + if (!refname) + die("symref-create: missing <ref>"); + + new_target = parse_next_refname(&next); + if (!new_target) + die("symref-create %s: missing <new-target>", refname); + + if (*next != line_termination) + die("symref-create %s: extra input: %s", refname, next); + + if (ref_transaction_create(transaction, refname, NULL, new_target, update_flags | create_reflog_flag, msg, &err)) die("%s", err.buf); update_flags = default_flags; free(refname); + free(new_target); strbuf_release(&err); } @@ -270,7 +413,7 @@ static void parse_cmd_delete(struct ref_transaction *transaction, if (ref_transaction_delete(transaction, refname, have_old ? &old_oid : NULL, - update_flags, msg, &err)) + NULL, update_flags, msg, &err)) die("%s", err.buf); update_flags = default_flags; @@ -278,6 +421,36 @@ static void parse_cmd_delete(struct ref_transaction *transaction, strbuf_release(&err); } + +static void parse_cmd_symref_delete(struct ref_transaction *transaction, + const char *next, const char *end) +{ + struct strbuf err = STRBUF_INIT; + char *refname, *old_target; + + if (!(update_flags & REF_NO_DEREF)) + die("symref-delete: cannot operate with deref mode"); + + refname = parse_refname(&next); + if (!refname) + die("symref-delete: missing <ref>"); + + old_target = parse_next_refname(&next); + + if (*next != line_termination) + die("symref-delete %s: extra input: %s", refname, next); + + if (ref_transaction_delete(transaction, refname, NULL, + old_target, update_flags, msg, &err)) + die("%s", err.buf); + + update_flags = default_flags; + free(refname); + free(old_target); + strbuf_release(&err); +} + + static void parse_cmd_verify(struct ref_transaction *transaction, const char *next, const char *end) { @@ -297,11 +470,47 @@ static void parse_cmd_verify(struct ref_transaction *transaction, die("verify %s: extra input: %s", refname, next); if (ref_transaction_verify(transaction, refname, &old_oid, - update_flags, &err)) + NULL, update_flags, &err)) + die("%s", err.buf); + + update_flags = default_flags; + free(refname); + strbuf_release(&err); +} + +static void parse_cmd_symref_verify(struct ref_transaction *transaction, + const char *next, const char *end) +{ + struct strbuf err = STRBUF_INIT; + struct object_id old_oid; + char *refname, *old_target; + + if (!(update_flags & REF_NO_DEREF)) + die("symref-verify: cannot operate with deref mode"); + + refname = parse_refname(&next); + if (!refname) + die("symref-verify: missing <ref>"); + + /* + * old_ref is optional, if not provided, we need to ensure that the + * ref doesn't exist. + */ + old_target = parse_next_refname(&next); + if (!old_target) + oidcpy(&old_oid, null_oid()); + + if (*next != line_termination) + die("symref-verify %s: extra input: %s", refname, next); + + if (ref_transaction_verify(transaction, refname, + old_target ? NULL : &old_oid, + old_target, update_flags, &err)) die("%s", err.buf); update_flags = default_flags; free(refname); + free(old_target); strbuf_release(&err); } @@ -380,15 +589,19 @@ static const struct parse_cmd { unsigned args; enum update_refs_state state; } command[] = { - { "update", parse_cmd_update, 3, UPDATE_REFS_OPEN }, - { "create", parse_cmd_create, 2, UPDATE_REFS_OPEN }, - { "delete", parse_cmd_delete, 2, UPDATE_REFS_OPEN }, - { "verify", parse_cmd_verify, 2, UPDATE_REFS_OPEN }, - { "option", parse_cmd_option, 1, UPDATE_REFS_OPEN }, - { "start", parse_cmd_start, 0, UPDATE_REFS_STARTED }, - { "prepare", parse_cmd_prepare, 0, UPDATE_REFS_PREPARED }, - { "abort", parse_cmd_abort, 0, UPDATE_REFS_CLOSED }, - { "commit", parse_cmd_commit, 0, UPDATE_REFS_CLOSED }, + { "update", parse_cmd_update, 3, UPDATE_REFS_OPEN }, + { "create", parse_cmd_create, 2, UPDATE_REFS_OPEN }, + { "delete", parse_cmd_delete, 2, UPDATE_REFS_OPEN }, + { "verify", parse_cmd_verify, 2, UPDATE_REFS_OPEN }, + { "symref-update", parse_cmd_symref_update, 4, UPDATE_REFS_OPEN }, + { "symref-create", parse_cmd_symref_create, 2, UPDATE_REFS_OPEN }, + { "symref-delete", parse_cmd_symref_delete, 2, UPDATE_REFS_OPEN }, + { "symref-verify", parse_cmd_symref_verify, 2, UPDATE_REFS_OPEN }, + { "option", parse_cmd_option, 1, UPDATE_REFS_OPEN }, + { "start", parse_cmd_start, 0, UPDATE_REFS_STARTED }, + { "prepare", parse_cmd_prepare, 0, UPDATE_REFS_PREPARED }, + { "abort", parse_cmd_abort, 0, UPDATE_REFS_CLOSED }, + { "commit", parse_cmd_commit, 0, UPDATE_REFS_CLOSED }, }; static void update_refs_stdin(void) diff --git a/commit-graph.c b/commit-graph.c index e5dd3553df..595a349c56 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1597,7 +1597,7 @@ static void compute_reachable_generation_numbers( timestamp_t gen; repo_parse_commit(info->r, c); gen = info->get_generation(c, info->data); - display_progress(info->progress, info->progress_cnt + 1); + display_progress(info->progress, ++info->progress_cnt); if (gen != GENERATION_NUMBER_ZERO && gen != GENERATION_NUMBER_INFINITY) continue; @@ -2002,8 +2002,8 @@ static int write_graph_chunk_base(struct hashfile *f, static int write_commit_graph_file(struct write_commit_graph_context *ctx) { uint32_t i; - int fd; struct hashfile *f; + struct tempfile *graph_layer; /* when ctx->split is non-zero */ struct lock_file lk = LOCK_INIT; const unsigned hashsz = the_hash_algo->rawsz; struct strbuf progress_title = STRBUF_INIT; @@ -2035,24 +2035,23 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) LOCK_DIE_ON_ERROR, 0444); free(lock_name); - fd = git_mkstemp_mode(ctx->graph_name, 0444); - if (fd < 0) { + graph_layer = mks_tempfile_m(ctx->graph_name, 0444); + if (!graph_layer) { error(_("unable to create temporary graph layer")); return -1; } - if (adjust_shared_perm(ctx->graph_name)) { + if (adjust_shared_perm(get_tempfile_path(graph_layer))) { error(_("unable to adjust shared permissions for '%s'"), - ctx->graph_name); + get_tempfile_path(graph_layer)); return -1; } - f = hashfd(fd, ctx->graph_name); + f = hashfd(get_tempfile_fd(graph_layer), get_tempfile_path(graph_layer)); } else { hold_lock_file_for_update_mode(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR, 0444); - fd = get_lock_file_fd(&lk); - f = hashfd(fd, get_lock_file_path(&lk)); + f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk)); } cf = init_chunkfile(f); @@ -2133,8 +2132,6 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) char *final_graph_name; int result; - close(fd); - if (!chainf) { error(_("unable to open commit-graph chain file")); return -1; @@ -2169,7 +2166,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) free(ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 1]); ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 1] = final_graph_name; - result = rename(ctx->graph_name, final_graph_name); + result = rename_tempfile(&graph_layer, final_graph_name); for (i = 0; i < ctx->num_commit_graphs_after; i++) fprintf(get_lock_file_fp(&lk), "%s\n", ctx->commit_graph_hash_after[i]); @@ -1244,6 +1244,15 @@ ssize_t git_config_ssize_t(const char *name, const char *value, return ret; } +double git_config_double(const char *name, const char *value, + const struct key_value_info *kvi) +{ + double ret; + if (!git_parse_double(value, &ret)) + die_bad_number(name, value, kvi); + return ret; +} + static const struct fsync_component_name { const char *name; enum fsync_component component_bits; @@ -1460,10 +1469,10 @@ static int git_default_core_config(const char *var, const char *value, if (!strcasecmp(value, "auto")) default_abbrev = -1; else if (!git_parse_maybe_bool_text(value)) - default_abbrev = the_hash_algo->hexsz; + default_abbrev = GIT_MAX_HEXSZ; else { int abbrev = git_config_int(var, value, ctx->kvi); - if (abbrev < minimum_abbrev || abbrev > the_hash_algo->hexsz) + if (abbrev < minimum_abbrev) return error(_("abbrev length out of range: %d"), abbrev); default_abbrev = abbrev; } @@ -262,6 +262,13 @@ ssize_t git_config_ssize_t(const char *, const char *, const struct key_value_info *); /** + * Identically to `git_config_double`, but for double-precision floating point + * values. + */ +double git_config_double(const char *, const char *, + const struct key_value_info *); + +/** * Same as `git_config_bool`, except that integers are returned as-is, and * an `is_bool` flag is unset. */ @@ -38,8 +38,8 @@ static int check_ref(const char *name, unsigned int flags) REFNAME_ALLOW_ONELEVEL)) return 0; - /* REF_HEADS means that we want regular branch heads */ - if ((flags & REF_HEADS) && starts_with(name, "heads/")) + /* REF_BRANCHES means that we want regular branch heads */ + if ((flags & REF_BRANCHES) && starts_with(name, "heads/")) return 1; /* REF_TAGS means that we want tags */ @@ -57,7 +57,7 @@ static int diff_color_moved_ws_default; static int diff_context_default = 3; static int diff_interhunk_context_default; static char *diff_word_regex_cfg; -static char *external_diff_cmd_cfg; +static struct external_diff external_diff_cfg; static char *diff_order_file_cfg; int diff_auto_refresh_index = 1; static int diff_mnemonic_prefix; @@ -431,7 +431,11 @@ int git_diff_ui_config(const char *var, const char *value, return 0; } if (!strcmp(var, "diff.external")) - return git_config_string(&external_diff_cmd_cfg, var, value); + return git_config_string(&external_diff_cfg.cmd, var, value); + if (!strcmp(var, "diff.trustexitcode")) { + external_diff_cfg.trust_exit_code = git_config_bool(var, value); + return 0; + } if (!strcmp(var, "diff.wordregex")) return git_config_string(&diff_word_regex_cfg, var, value); if (!strcmp(var, "diff.orderfile")) @@ -548,18 +552,22 @@ static char *quote_two(const char *one, const char *two) return strbuf_detach(&res, NULL); } -static const char *external_diff(void) +static const struct external_diff *external_diff(void) { - static const char *external_diff_cmd = NULL; + static struct external_diff external_diff_env, *external_diff_ptr; static int done_preparing = 0; if (done_preparing) - return external_diff_cmd; - external_diff_cmd = xstrdup_or_null(getenv("GIT_EXTERNAL_DIFF")); - if (!external_diff_cmd) - external_diff_cmd = external_diff_cmd_cfg; + return external_diff_ptr; + external_diff_env.cmd = xstrdup_or_null(getenv("GIT_EXTERNAL_DIFF")); + if (git_env_bool("GIT_EXTERNAL_DIFF_TRUST_EXIT_CODE", 0)) + external_diff_env.trust_exit_code = 1; + if (external_diff_env.cmd) + external_diff_ptr = &external_diff_env; + else if (external_diff_cfg.cmd) + external_diff_ptr = &external_diff_cfg; done_preparing = 1; - return external_diff_cmd; + return external_diff_ptr; } /* @@ -4375,7 +4383,7 @@ static void add_external_diff_name(struct repository *r, * infile2 infile2-sha1 infile2-mode [ rename-to ] * */ -static void run_external_diff(const char *pgm, +static void run_external_diff(const struct external_diff *pgm, const char *name, const char *other, struct diff_filespec *one, @@ -4385,8 +4393,21 @@ static void run_external_diff(const char *pgm, { struct child_process cmd = CHILD_PROCESS_INIT; struct diff_queue_struct *q = &diff_queued_diff; + int quiet = !(o->output_format & DIFF_FORMAT_PATCH); + int rc; + + /* + * Trivial equality is handled by diff_unmodified_pair() before + * we get here. If we don't need to show the diff and the + * external diff program lacks the ability to tell us whether + * it's empty then we consider it non-empty without even asking. + */ + if (!pgm->trust_exit_code && quiet) { + o->found_changes = 1; + return; + } - strvec_push(&cmd.args, pgm); + strvec_push(&cmd.args, pgm->cmd); strvec_push(&cmd.args, name); if (one && two) { @@ -4406,7 +4427,15 @@ static void run_external_diff(const char *pgm, diff_free_filespec_data(one); diff_free_filespec_data(two); cmd.use_shell = 1; - if (run_command(&cmd)) + cmd.no_stdout = quiet; + rc = run_command(&cmd); + if (!pgm->trust_exit_code && rc == 0) + o->found_changes = 1; + else if (pgm->trust_exit_code && rc == 0) + ; /* nothing */ + else if (pgm->trust_exit_code && rc == 1) + o->found_changes = 1; + else die(_("external diff died, stopping at %s"), name); remove_tempfile(); @@ -4512,7 +4541,7 @@ static void fill_metainfo(struct strbuf *msg, } } -static void run_diff_cmd(const char *pgm, +static void run_diff_cmd(const struct external_diff *pgm, const char *name, const char *other, const char *attr_path, @@ -4530,8 +4559,8 @@ static void run_diff_cmd(const char *pgm, if (o->flags.allow_external || !o->ignore_driver_algorithm) drv = userdiff_find_by_path(o->repo->index, attr_path); - if (o->flags.allow_external && drv && drv->external) - pgm = drv->external; + if (o->flags.allow_external && drv && drv->external.cmd) + pgm = &drv->external; if (msg) { /* @@ -4597,7 +4626,7 @@ static void strip_prefix(int prefix_length, const char **namep, const char **oth static void run_diff(struct diff_filepair *p, struct diff_options *o) { - const char *pgm = external_diff(); + const struct external_diff *pgm = external_diff(); struct strbuf msg; struct diff_filespec *one = p->one; struct diff_filespec *two = p->two; @@ -4924,6 +4953,13 @@ void diff_setup_done(struct diff_options *options) options->flags.exit_with_status = 1; } + /* + * External diffs could declare non-identical contents equal + * (think diff --ignore-space-change). + */ + if (options->flags.allow_external && options->flags.exit_with_status) + options->flags.diff_from_contents = 1; + options->diff_path_counter = 0; if (options->flags.follow_renames) diff --git a/ewah/bitmap.c b/ewah/bitmap.c index ac7e0af622..55928dada8 100644 --- a/ewah/bitmap.c +++ b/ewah/bitmap.c @@ -138,6 +138,49 @@ void bitmap_or(struct bitmap *self, const struct bitmap *other) self->words[i] |= other->words[i]; } +int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other) +{ + struct ewah_iterator it; + eword_t word; + size_t i; + + ewah_iterator_init(&it, self); + + for (i = 0; i < other->word_alloc; i++) { + if (!ewah_iterator_next(&word, &it)) { + /* + * If we reached the end of `self`, and haven't + * rejected `self` as a possible subset of + * `other` yet, then we are done and `self` is + * indeed a subset of `other`. + */ + return 1; + } + if (word & ~other->words[i]) { + /* + * Otherwise, compare the next two pairs of + * words. If the word from `self` has bit(s) not + * in the word from `other`, `self` is not a + * subset of `other`. + */ + return 0; + } + } + + /* + * If we got to this point, there may be zero or more words + * remaining in `self`, with no remaining words left in `other`. + * If there are any bits set in the remaining word(s) in `self`, + * then `self` is not a subset of `other`. + */ + while (ewah_iterator_next(&word, &it)) + if (word) + return 0; + + /* `self` is definitely a subset of `other` */ + return 1; +} + void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other) { size_t original_size = self->word_alloc; @@ -169,6 +212,20 @@ size_t bitmap_popcount(struct bitmap *self) return count; } +size_t ewah_bitmap_popcount(struct ewah_bitmap *self) +{ + struct ewah_iterator it; + eword_t word; + size_t count = 0; + + ewah_iterator_init(&it, self); + + while (ewah_iterator_next(&word, &it)) + count += ewah_bit_popcount64(word); + + return count; +} + int bitmap_is_empty(struct bitmap *self) { size_t i; @@ -204,6 +261,25 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other) return 1; } +int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other) +{ + struct ewah_iterator it; + eword_t word; + size_t i = 0; + + ewah_iterator_init(&it, other); + + while (ewah_iterator_next(&word, &it)) + if (word != (i < self->word_alloc ? self->words[i++] : 0)) + return 0; + + for (; i < self->word_alloc; i++) + if (self->words[i]) + return 0; + + return 1; +} + int bitmap_is_subset(struct bitmap *self, struct bitmap *other) { size_t common_size, i; diff --git a/ewah/ewok.h b/ewah/ewok.h index c11d76c6f3..5e357e2493 100644 --- a/ewah/ewok.h +++ b/ewah/ewok.h @@ -179,7 +179,14 @@ void bitmap_unset(struct bitmap *self, size_t pos); int bitmap_get(struct bitmap *self, size_t pos); void bitmap_free(struct bitmap *self); int bitmap_equals(struct bitmap *self, struct bitmap *other); +int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other); + +/* + * Both `bitmap_is_subset()` and `ewah_bitmap_is_subset()` return 1 if the set + * of bits in 'self' are a subset of the bits in 'other'. Returns 0 otherwise. + */ int bitmap_is_subset(struct bitmap *self, struct bitmap *other); +int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other); struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap); struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah); @@ -189,6 +196,7 @@ void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other); void bitmap_or(struct bitmap *self, const struct bitmap *other); size_t bitmap_popcount(struct bitmap *self); +size_t ewah_bitmap_popcount(struct ewah_bitmap *self); int bitmap_is_empty(struct bitmap *self); #endif diff --git a/midx-write.c b/midx-write.c index 55a6b63bac..a941e363b7 100644 --- a/midx-write.c +++ b/midx-write.c @@ -101,13 +101,27 @@ struct write_midx_context { }; static int should_include_pack(const struct write_midx_context *ctx, - const char *file_name, - int exclude_from_midx) + const char *file_name) { - if (exclude_from_midx && ctx->m && midx_contains_pack(ctx->m, file_name)) + /* + * Note that at most one of ctx->m and ctx->to_include are set, + * so we are testing midx_contains_pack() and + * string_list_has_string() independently (guarded by the + * appropriate NULL checks). + * + * We could support passing to_include while reusing an existing + * MIDX, but don't currently since the reuse process drags + * forward all packs from an existing MIDX (without checking + * whether or not they appear in the to_include list). + * + * If we added support for that, these next two conditional + * should be performed independently (likely checking + * to_include before the existing MIDX). + */ + if (ctx->m && midx_contains_pack(ctx->m, file_name)) return 0; - if (ctx->to_include && !string_list_has_string(ctx->to_include, - file_name)) + else if (ctx->to_include && + !string_list_has_string(ctx->to_include, file_name)) return 0; return 1; } @@ -121,7 +135,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, if (ends_with(file_name, ".idx")) { display_progress(ctx->progress, ++ctx->pack_paths_checked); - if (!should_include_pack(ctx, file_name, 1)) + if (!should_include_pack(ctx, file_name)) return; ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); @@ -809,7 +823,7 @@ static int write_midx_bitmap(const char *midx_name, for (i = 0; i < pdata->nr_objects; i++) index[i] = &pdata->objects[i].idx; - bitmap_writer_init(&writer); + bitmap_writer_init(&writer, the_repository); bitmap_writer_show_progress(&writer, flags & MIDX_PROGRESS); bitmap_writer_build_type_index(&writer, pdata, index, pdata->nr_objects); @@ -880,9 +894,6 @@ static int fill_packs_from_midx(struct write_midx_context *ctx, uint32_t i; for (i = 0; i < ctx->m->num_packs; i++) { - if (!should_include_pack(ctx, ctx->m->pack_names[i], 0)) - continue; - ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); if (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) { @@ -937,7 +948,15 @@ static int write_midx_internal(const char *object_dir, die_errno(_("unable to create leading directories of %s"), midx_name.buf); - ctx.m = lookup_multi_pack_index(the_repository, object_dir); + if (!packs_to_include) { + /* + * Only reference an existing MIDX when not filtering which + * packs to include, since all packs and objects are copied + * blindly from an existing MIDX if one is present. + */ + ctx.m = lookup_multi_pack_index(the_repository, object_dir); + } + if (ctx.m && !midx_checksum_valid(ctx.m)) { warning(_("ignoring existing multi-pack-index; checksum mismatch")); ctx.m = NULL; @@ -946,7 +965,6 @@ static int write_midx_internal(const char *object_dir, ctx.nr = 0; ctx.alloc = ctx.m ? ctx.m->num_packs : 16; ctx.info = NULL; - ctx.to_include = packs_to_include; ALLOC_ARRAY(ctx.info, ctx.alloc); if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name, @@ -963,6 +981,8 @@ static int write_midx_internal(const char *object_dir, else ctx.progress = NULL; + ctx.to_include = packs_to_include; + for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx); stop_progress(&ctx.progress); diff --git a/object-name.c b/object-name.c index 523af6f64f..1be2ad1a16 100644 --- a/object-name.c +++ b/object-name.c @@ -837,7 +837,7 @@ int repo_find_unique_abbrev_r(struct repository *r, char *hex, } oid_to_hex_r(hex, oid); - if (len == hexsz || !len) + if (len >= hexsz || !len) return hexsz; mad.repo = r; @@ -81,7 +81,7 @@ void object_array_init(struct object_array *array); * reflog.c: 10--12 * builtin/show-branch.c: 0-------------------------------------------26 * builtin/unpack-objects.c: 2021 - * pack-bitmap.h: 22 + * pack-bitmap.h: 2122 */ #define FLAG_BITS 28 diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index 6cae670412..6e8060f8a0 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -17,6 +17,12 @@ #include "trace2.h" #include "tree.h" #include "tree-walk.h" +#include "pseudo-merge.h" +#include "oid-array.h" +#include "config.h" +#include "alloc.h" +#include "refs.h" +#include "strmap.h" struct bitmapped_commit { struct commit *commit; @@ -25,16 +31,39 @@ struct bitmapped_commit { int flags; int xor_offset; uint32_t commit_pos; + unsigned pseudo_merge : 1; }; -void bitmap_writer_init(struct bitmap_writer *writer) +static inline int bitmap_writer_nr_selected_commits(struct bitmap_writer *writer) +{ + return writer->selected_nr - writer->pseudo_merges_nr; +} + +void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r) { memset(writer, 0, sizeof(struct bitmap_writer)); + if (writer->bitmaps) + BUG("bitmap writer already initialized"); + writer->bitmaps = kh_init_oid_map(); + writer->pseudo_merge_commits = kh_init_oid_map(); + + string_list_init_dup(&writer->pseudo_merge_groups); + + load_pseudo_merges_from_config(&writer->pseudo_merge_groups); +} + +static void free_pseudo_merge_commit_idx(struct pseudo_merge_commit_idx *idx) +{ + if (!idx) + return; + free(idx->pseudo_merge); + free(idx); } void bitmap_writer_free(struct bitmap_writer *writer) { uint32_t i; + struct pseudo_merge_commit_idx *idx; if (!writer) return; @@ -46,6 +75,10 @@ void bitmap_writer_free(struct bitmap_writer *writer) kh_destroy_oid_map(writer->bitmaps); + kh_foreach_value(writer->pseudo_merge_commits, idx, + free_pseudo_merge_commit_idx(idx)); + kh_destroy_oid_map(writer->pseudo_merge_commits); + for (i = 0; i < writer->selected_nr; i++) { struct bitmapped_commit *bc = &writer->selected[i]; if (bc->write_as != bc->bitmap) @@ -121,22 +154,41 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer, } } +int bitmap_writer_has_bitmapped_object_id(struct bitmap_writer *writer, + const struct object_id *oid) +{ + return kh_get_oid_map(writer->bitmaps, *oid) != kh_end(writer->bitmaps); +} + /** * Compute the actual bitmaps */ -static inline void push_bitmapped_commit(struct bitmap_writer *writer, - struct commit *commit) +void bitmap_writer_push_commit(struct bitmap_writer *writer, + struct commit *commit, unsigned pseudo_merge) { if (writer->selected_nr >= writer->selected_alloc) { writer->selected_alloc = (writer->selected_alloc + 32) * 2; REALLOC_ARRAY(writer->selected, writer->selected_alloc); } + if (!pseudo_merge) { + int hash_ret; + khiter_t hash_pos = kh_put_oid_map(writer->bitmaps, + commit->object.oid, + &hash_ret); + + if (!hash_ret) + die(_("duplicate entry when writing bitmap index: %s"), + oid_to_hex(&commit->object.oid)); + kh_value(writer->bitmaps, hash_pos) = NULL; + } + writer->selected[writer->selected_nr].commit = commit; writer->selected[writer->selected_nr].bitmap = NULL; writer->selected[writer->selected_nr].write_as = NULL; writer->selected[writer->selected_nr].flags = 0; + writer->selected[writer->selected_nr].pseudo_merge = pseudo_merge; writer->selected_nr++; } @@ -167,16 +219,20 @@ static void compute_xor_offsets(struct bitmap_writer *writer) while (next < writer->selected_nr) { struct bitmapped_commit *stored = &writer->selected[next]; - int best_offset = 0; struct ewah_bitmap *best_bitmap = stored->bitmap; struct ewah_bitmap *test_xor; + if (stored->pseudo_merge) + goto next; + for (i = 1; i <= MAX_XOR_OFFSET_SEARCH; ++i) { int curr = next - i; if (curr < 0) break; + if (writer->selected[curr].pseudo_merge) + continue; test_xor = ewah_pool_new(); ewah_xor(writer->selected[curr].bitmap, stored->bitmap, test_xor); @@ -192,6 +248,7 @@ static void compute_xor_offsets(struct bitmap_writer *writer) } } +next: stored->xor_offset = best_offset; stored->write_as = best_bitmap; @@ -204,7 +261,8 @@ struct bb_commit { struct bitmap *commit_mask; struct bitmap *bitmap; unsigned selected:1, - maximal:1; + maximal:1, + pseudo_merge:1; unsigned idx; /* within selected array */ }; @@ -242,17 +300,18 @@ static void bitmap_builder_init(struct bitmap_builder *bb, revs.first_parent_only = 1; for (i = 0; i < writer->selected_nr; i++) { - struct commit *c = writer->selected[i].commit; - struct bb_commit *ent = bb_data_at(&bb->data, c); + struct bitmapped_commit *bc = &writer->selected[i]; + struct bb_commit *ent = bb_data_at(&bb->data, bc->commit); ent->selected = 1; ent->maximal = 1; + ent->pseudo_merge = bc->pseudo_merge; ent->idx = i; ent->commit_mask = bitmap_new(); bitmap_set(ent->commit_mask, i); - add_pending_object(&revs, &c->object, ""); + add_pending_object(&revs, &bc->commit->object, ""); } if (prepare_revision_walk(&revs)) @@ -410,6 +469,7 @@ static int fill_bitmap_tree(struct bitmap_writer *writer, } static int reused_bitmaps_nr; +static int reused_pseudo_merge_bitmaps_nr; static int fill_bitmap_commit(struct bitmap_writer *writer, struct bb_commit *ent, @@ -431,8 +491,13 @@ static int fill_bitmap_commit(struct bitmap_writer *writer, struct commit *c = prio_queue_get(queue); if (old_bitmap && mapping) { - struct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c); + struct ewah_bitmap *old; struct bitmap *remapped = bitmap_new(); + + if (commit->object.flags & BITMAP_PSEUDO_MERGE) + old = pseudo_merge_bitmap_for_commit(old_bitmap, c); + else + old = bitmap_for_commit(old_bitmap, c); /* * If this commit has an old bitmap, then translate that * bitmap and add its bits to this one. No need to walk @@ -441,7 +506,10 @@ static int fill_bitmap_commit(struct bitmap_writer *writer, if (old && !rebuild_bitmap(mapping, old, remapped)) { bitmap_or(ent->bitmap, remapped); bitmap_free(remapped); - reused_bitmaps_nr++; + if (commit->object.flags & BITMAP_PSEUDO_MERGE) + reused_pseudo_merge_bitmaps_nr++; + else + reused_bitmaps_nr++; continue; } bitmap_free(remapped); @@ -451,12 +519,14 @@ static int fill_bitmap_commit(struct bitmap_writer *writer, * Mark ourselves and queue our tree. The commit * walk ensures we cover all parents. */ - pos = find_object_pos(writer, &c->object.oid, &found); - if (!found) - return -1; - bitmap_set(ent->bitmap, pos); - prio_queue_put(tree_queue, - repo_get_commit_tree(the_repository, c)); + if (!(c->object.flags & BITMAP_PSEUDO_MERGE)) { + pos = find_object_pos(writer, &c->object.oid, &found); + if (!found) + return -1; + bitmap_set(ent->bitmap, pos); + prio_queue_put(tree_queue, + repo_get_commit_tree(the_repository, c)); + } for (p = c->parents; p; p = p->next) { pos = find_object_pos(writer, &p->item->object.oid, @@ -483,14 +553,17 @@ static void store_selected(struct bitmap_writer *writer, { struct bitmapped_commit *stored = &writer->selected[ent->idx]; khiter_t hash_pos; - int hash_ret; stored->bitmap = bitmap_to_ewah(ent->bitmap); - hash_pos = kh_put_oid_map(writer->bitmaps, commit->object.oid, &hash_ret); - if (hash_ret == 0) - die("Duplicate entry when writing index: %s", + if (ent->pseudo_merge) + return; + + hash_pos = kh_get_oid_map(writer->bitmaps, commit->object.oid); + if (hash_pos == kh_end(writer->bitmaps)) + die(_("attempted to store non-selected commit: '%s'"), oid_to_hex(&commit->object.oid)); + kh_value(writer->bitmaps, hash_pos) = stored; } @@ -506,7 +579,6 @@ int bitmap_writer_build(struct bitmap_writer *writer, uint32_t *mapping; int closed = 1; /* until proven otherwise */ - writer->bitmaps = kh_init_oid_map(); writer->to_pack = to_pack; if (writer->show_progress) @@ -567,6 +639,9 @@ int bitmap_writer_build(struct bitmap_writer *writer, the_repository); trace2_data_intmax("pack-bitmap-write", the_repository, "building_bitmaps_reused", reused_bitmaps_nr); + trace2_data_intmax("pack-bitmap-write", the_repository, + "building_bitmaps_pseudo_merge_reused", + reused_pseudo_merge_bitmaps_nr); stop_progress(&writer->progress); @@ -619,7 +694,7 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer, if (indexed_commits_nr < 100) { for (i = 0; i < indexed_commits_nr; ++i) - push_bitmapped_commit(writer, indexed_commits[i]); + bitmap_writer_push_commit(writer, indexed_commits[i], 0); return; } @@ -652,13 +727,15 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer, } } - push_bitmapped_commit(writer, chosen); + bitmap_writer_push_commit(writer, chosen, 0); i += next + 1; display_progress(writer->progress, i); } stop_progress(&writer->progress); + + select_pseudo_merges(writer, indexed_commits, indexed_commits_nr); } @@ -689,8 +766,11 @@ static void write_selected_commits_v1(struct bitmap_writer *writer, { int i; - for (i = 0; i < writer->selected_nr; ++i) { + for (i = 0; i < bitmap_writer_nr_selected_commits(writer); ++i) { struct bitmapped_commit *stored = &writer->selected[i]; + if (stored->pseudo_merge) + BUG("unexpected pseudo-merge among selected: %s", + oid_to_hex(&stored->commit->object.oid)); if (offsets) offsets[i] = hashfile_total(f); @@ -703,6 +783,130 @@ static void write_selected_commits_v1(struct bitmap_writer *writer, } } +static void write_pseudo_merges(struct bitmap_writer *writer, + struct hashfile *f) +{ + struct oid_array commits = OID_ARRAY_INIT; + struct bitmap **commits_bitmap = NULL; + off_t *pseudo_merge_ofs = NULL; + off_t start, table_start, next_ext; + + uint32_t base = bitmap_writer_nr_selected_commits(writer); + size_t i, j = 0; + + CALLOC_ARRAY(commits_bitmap, writer->pseudo_merges_nr); + CALLOC_ARRAY(pseudo_merge_ofs, writer->pseudo_merges_nr); + + for (i = 0; i < writer->pseudo_merges_nr; i++) { + struct bitmapped_commit *merge = &writer->selected[base + i]; + struct commit_list *p; + + if (!merge->pseudo_merge) + BUG("found non-pseudo merge commit at %"PRIuMAX, (uintmax_t)i); + + commits_bitmap[i] = bitmap_new(); + + for (p = merge->commit->parents; p; p = p->next) + bitmap_set(commits_bitmap[i], + find_object_pos(writer, &p->item->object.oid, + NULL)); + } + + start = hashfile_total(f); + + for (i = 0; i < writer->pseudo_merges_nr; i++) { + struct ewah_bitmap *commits_ewah = bitmap_to_ewah(commits_bitmap[i]); + + pseudo_merge_ofs[i] = hashfile_total(f); + + dump_bitmap(f, commits_ewah); + dump_bitmap(f, writer->selected[base+i].write_as); + + ewah_free(commits_ewah); + } + + next_ext = st_add(hashfile_total(f), + st_mult(kh_size(writer->pseudo_merge_commits), + sizeof(uint64_t))); + + table_start = hashfile_total(f); + + commits.alloc = kh_size(writer->pseudo_merge_commits); + CALLOC_ARRAY(commits.oid, commits.alloc); + + for (i = kh_begin(writer->pseudo_merge_commits); i != kh_end(writer->pseudo_merge_commits); i++) { + if (!kh_exist(writer->pseudo_merge_commits, i)) + continue; + oid_array_append(&commits, &kh_key(writer->pseudo_merge_commits, i)); + } + + oid_array_sort(&commits); + + /* write lookup table (non-extended) */ + for (i = 0; i < commits.nr; i++) { + int hash_pos; + struct pseudo_merge_commit_idx *c; + + hash_pos = kh_get_oid_map(writer->pseudo_merge_commits, + commits.oid[i]); + if (hash_pos == kh_end(writer->pseudo_merge_commits)) + BUG("could not find pseudo-merge commit %s", + oid_to_hex(&commits.oid[i])); + + c = kh_value(writer->pseudo_merge_commits, hash_pos); + + hashwrite_be32(f, find_object_pos(writer, &commits.oid[i], + NULL)); + if (c->nr == 1) + hashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[0]]); + else if (c->nr > 1) { + if (next_ext & ((uint64_t)1<<63)) + die(_("too many pseudo-merges")); + hashwrite_be64(f, next_ext | ((uint64_t)1<<63)); + next_ext = st_add3(next_ext, + sizeof(uint32_t), + st_mult(c->nr, sizeof(uint64_t))); + } else + BUG("expected commit '%s' to have at least one " + "pseudo-merge", oid_to_hex(&commits.oid[i])); + } + + /* write lookup table (extended) */ + for (i = 0; i < commits.nr; i++) { + int hash_pos; + struct pseudo_merge_commit_idx *c; + + hash_pos = kh_get_oid_map(writer->pseudo_merge_commits, + commits.oid[i]); + if (hash_pos == kh_end(writer->pseudo_merge_commits)) + BUG("could not find pseudo-merge commit %s", + oid_to_hex(&commits.oid[i])); + + c = kh_value(writer->pseudo_merge_commits, hash_pos); + if (c->nr == 1) + continue; + + hashwrite_be32(f, c->nr); + for (j = 0; j < c->nr; j++) + hashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[j]]); + } + + /* write positions for all pseudo merges */ + for (i = 0; i < writer->pseudo_merges_nr; i++) + hashwrite_be64(f, pseudo_merge_ofs[i]); + + hashwrite_be32(f, writer->pseudo_merges_nr); + hashwrite_be32(f, kh_size(writer->pseudo_merge_commits)); + hashwrite_be64(f, table_start - start); + hashwrite_be64(f, hashfile_total(f) - start + sizeof(uint64_t)); + + for (i = 0; i < writer->pseudo_merges_nr; i++) + bitmap_free(commits_bitmap[i]); + + free(pseudo_merge_ofs); + free(commits_bitmap); +} + static int table_cmp(const void *_va, const void *_vb, void *_data) { struct bitmap_writer *writer = _data; @@ -723,10 +927,10 @@ static void write_lookup_table(struct bitmap_writer *writer, struct hashfile *f, uint32_t i; uint32_t *table, *table_inv; - ALLOC_ARRAY(table, writer->selected_nr); - ALLOC_ARRAY(table_inv, writer->selected_nr); + ALLOC_ARRAY(table, bitmap_writer_nr_selected_commits(writer)); + ALLOC_ARRAY(table_inv, bitmap_writer_nr_selected_commits(writer)); - for (i = 0; i < writer->selected_nr; i++) + for (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) table[i] = i; /* @@ -734,16 +938,16 @@ static void write_lookup_table(struct bitmap_writer *writer, struct hashfile *f, * bitmap corresponds to j'th bitmapped commit (among the selected * commits) in lex order of OIDs. */ - QSORT_S(table, writer->selected_nr, table_cmp, writer); + QSORT_S(table, bitmap_writer_nr_selected_commits(writer), table_cmp, writer); /* table_inv helps us discover that relationship (i'th bitmap * to j'th commit by j = table_inv[i]) */ - for (i = 0; i < writer->selected_nr; i++) + for (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) table_inv[table[i]] = i; trace2_region_enter("pack-bitmap-write", "writing_lookup_table", the_repository); - for (i = 0; i < writer->selected_nr; i++) { + for (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) { struct bitmapped_commit *selected = &writer->selected[table[i]]; uint32_t xor_offset = selected->xor_offset; uint32_t xor_row; @@ -810,12 +1014,15 @@ void bitmap_writer_finish(struct bitmap_writer *writer, int fd = odb_mkstemp(&tmp_file, "pack/tmp_bitmap_XXXXXX"); + if (writer->pseudo_merges_nr) + options |= BITMAP_OPT_PSEUDO_MERGES; + f = hashfd(fd, tmp_file.buf); memcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE)); header.version = htons(default_version); header.options = htons(flags | options); - header.entry_count = htonl(writer->selected_nr); + header.entry_count = htonl(bitmap_writer_nr_selected_commits(writer)); hashcpy(header.checksum, writer->pack_checksum); hashwrite(f, &header, sizeof(header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz); @@ -827,7 +1034,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer, if (options & BITMAP_OPT_LOOKUP_TABLE) CALLOC_ARRAY(offsets, index_nr); - for (i = 0; i < writer->selected_nr; i++) { + for (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) { struct bitmapped_commit *stored = &writer->selected[i]; int commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access); @@ -839,6 +1046,9 @@ void bitmap_writer_finish(struct bitmap_writer *writer, write_selected_commits_v1(writer, f, offsets); + if (options & BITMAP_OPT_PSEUDO_MERGES) + write_pseudo_merges(writer, f); + if (options & BITMAP_OPT_LOOKUP_TABLE) write_lookup_table(writer, f, offsets); diff --git a/pack-bitmap.c b/pack-bitmap.c index fe8e8a51d3..afe41f6ba6 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -20,6 +20,7 @@ #include "list-objects-filter-options.h" #include "midx.h" #include "config.h" +#include "pseudo-merge.h" /* * An entry on the bitmap index, representing the bitmap for a given @@ -86,6 +87,9 @@ struct bitmap_index { */ unsigned char *table_lookup; + /* This contains the pseudo-merge cache within 'map' (if found). */ + struct pseudo_merge_map pseudo_merges; + /* * Extended index. * @@ -110,6 +114,13 @@ struct bitmap_index { unsigned int version; }; +static int pseudo_merges_satisfied_nr; +static int pseudo_merges_cascades_nr; +static int existing_bitmaps_hits_nr; +static int existing_bitmaps_misses_nr; +static int roots_with_bitmaps_nr; +static int roots_without_bitmaps_nr; + static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st) { struct ewah_bitmap *parent; @@ -129,17 +140,13 @@ static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st) return composed; } -/* - * Read a bitmap from the current read position on the mmaped - * index, and increase the read position accordingly - */ -static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index) +struct ewah_bitmap *read_bitmap(const unsigned char *map, + size_t map_size, size_t *map_pos) { struct ewah_bitmap *b = ewah_pool_new(); - ssize_t bitmap_size = ewah_read_mmap(b, - index->map + index->map_pos, - index->map_size - index->map_pos); + ssize_t bitmap_size = ewah_read_mmap(b, map + *map_pos, + map_size - *map_pos); if (bitmap_size < 0) { error(_("failed to load bitmap index (corrupted?)")); @@ -147,10 +154,20 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index) return NULL; } - index->map_pos += bitmap_size; + *map_pos += bitmap_size; + return b; } +/* + * Read a bitmap from the current read position on the mmaped + * index, and increase the read position accordingly + */ +static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index) +{ + return read_bitmap(index->map, index->map_size, &index->map_pos); +} + static uint32_t bitmap_num_objects(struct bitmap_index *index) { if (index->midx) @@ -199,6 +216,46 @@ static int load_bitmap_header(struct bitmap_index *index) index->table_lookup = (void *)(index_end - table_size); index_end -= table_size; } + + if (flags & BITMAP_OPT_PSEUDO_MERGES) { + unsigned char *pseudo_merge_ofs; + size_t table_size; + uint32_t i; + + if (sizeof(table_size) > index_end - index->map - header_size) + return error(_("corrupted bitmap index file (too short to fit pseudo-merge table header)")); + + table_size = get_be64(index_end - 8); + if (table_size > index_end - index->map - header_size) + return error(_("corrupted bitmap index file (too short to fit pseudo-merge table)")); + + if (git_env_bool("GIT_TEST_USE_PSEUDO_MERGES", 1)) { + const unsigned char *ext = (index_end - table_size); + + index->pseudo_merges.map = index->map; + index->pseudo_merges.map_size = index->map_size; + index->pseudo_merges.commits = ext + get_be64(index_end - 16); + index->pseudo_merges.commits_nr = get_be32(index_end - 20); + index->pseudo_merges.nr = get_be32(index_end - 24); + + if (st_add(st_mult(index->pseudo_merges.nr, + sizeof(uint64_t)), + 24) > table_size) + return error(_("corrupted bitmap index file, pseudo-merge table too short")); + + CALLOC_ARRAY(index->pseudo_merges.v, + index->pseudo_merges.nr); + + pseudo_merge_ofs = index_end - 24 - + (index->pseudo_merges.nr * sizeof(uint64_t)); + for (i = 0; i < index->pseudo_merges.nr; i++) { + index->pseudo_merges.v[i].at = get_be64(pseudo_merge_ofs); + pseudo_merge_ofs += sizeof(uint64_t); + } + } + + index_end -= table_size; + } } index->entry_count = ntohl(header->entry_count); @@ -960,6 +1017,22 @@ static void show_commit(struct commit *commit UNUSED, { } +static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git, + struct bitmap *result, + struct commit *commit, + uint32_t commit_pos) +{ + int ret; + + ret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges, + result, commit, commit_pos); + + if (ret) + pseudo_merges_satisfied_nr += ret; + + return ret; +} + static int add_to_include_set(struct bitmap_index *bitmap_git, struct include_data *data, struct commit *commit, @@ -975,11 +1048,19 @@ static int add_to_include_set(struct bitmap_index *bitmap_git, partial = bitmap_for_commit(bitmap_git, commit); if (partial) { + existing_bitmaps_hits_nr++; + bitmap_or_ewah(data->base, partial); return 0; } + existing_bitmaps_misses_nr++; + bitmap_set(data->base, bitmap_pos); + if (apply_pseudo_merges_for_commit_1(bitmap_git, data->base, commit, + bitmap_pos)) + return 0; + return 1; } @@ -1030,8 +1111,12 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git, { struct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit); - if (!or_with) + if (!or_with) { + existing_bitmaps_misses_nr++; return 0; + } + + existing_bitmaps_hits_nr++; if (!*base) *base = ewah_to_bitmap(or_with); @@ -1105,6 +1190,20 @@ static void show_boundary_object(struct object *object UNUSED, BUG("should not be called"); } +static unsigned cascade_pseudo_merges_1(struct bitmap_index *bitmap_git, + struct bitmap *result, + struct bitmap *roots) +{ + int ret = cascade_pseudo_merges(&bitmap_git->pseudo_merges, + result, roots); + if (ret) { + pseudo_merges_cascades_nr++; + pseudo_merges_satisfied_nr += ret; + } + + return ret; +} + static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git, struct rev_info *revs, struct object_list *roots) @@ -1114,6 +1213,7 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git, unsigned int i; unsigned int tmp_blobs, tmp_trees, tmp_tags; int any_missing = 0; + int existing_bitmaps = 0; cb.bitmap_git = bitmap_git; cb.base = bitmap_new(); @@ -1121,6 +1221,25 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git, revs->ignore_missing_links = 1; + if (bitmap_git->pseudo_merges.nr) { + struct bitmap *roots_bitmap = bitmap_new(); + struct object_list *objects = NULL; + + for (objects = roots; objects; objects = objects->next) { + struct object *object = objects->item; + int pos; + + pos = bitmap_position(bitmap_git, &object->oid); + if (pos < 0) + continue; + + bitmap_set(roots_bitmap, pos); + } + + if (!cascade_pseudo_merges_1(bitmap_git, cb.base, roots_bitmap)) + bitmap_free(roots_bitmap); + } + /* * OR in any existing reachability bitmaps among `roots` into * `cb.base`. @@ -1132,8 +1251,10 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git, continue; if (add_commit_to_bitmap(bitmap_git, &cb.base, - (struct commit *)object)) + (struct commit *)object)) { + existing_bitmaps = 1; continue; + } any_missing = 1; } @@ -1141,6 +1262,9 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git, if (!any_missing) goto cleanup; + if (existing_bitmaps) + cascade_pseudo_merges_1(bitmap_git, cb.base, NULL); + tmp_blobs = revs->blob_objects; tmp_trees = revs->tree_objects; tmp_tags = revs->blob_objects; @@ -1196,6 +1320,44 @@ cleanup: return cb.base; } +struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git, + struct commit *commit) +{ + struct commit_list *p; + struct bitmap *parents; + struct pseudo_merge *match = NULL; + + if (!bitmap_git->pseudo_merges.nr) + return NULL; + + parents = bitmap_new(); + + for (p = commit->parents; p; p = p->next) { + int pos = bitmap_position(bitmap_git, &p->item->object.oid); + if (pos < 0 || pos >= bitmap_num_objects(bitmap_git)) + goto done; + + bitmap_set(parents, pos); + } + + match = pseudo_merge_for_parents(&bitmap_git->pseudo_merges, + parents); + +done: + bitmap_free(parents); + if (match) + return pseudo_merge_bitmap(&bitmap_git->pseudo_merges, match); + + return NULL; +} + +static void unsatisfy_all_pseudo_merges(struct bitmap_index *bitmap_git) +{ + uint32_t i; + for (i = 0; i < bitmap_git->pseudo_merges.nr; i++) + bitmap_git->pseudo_merges.v[i].satisfied = 0; +} + static struct bitmap *find_objects(struct bitmap_index *bitmap_git, struct rev_info *revs, struct object_list *roots, @@ -1203,9 +1365,32 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git, { struct bitmap *base = NULL; int needs_walk = 0; + unsigned existing_bitmaps = 0; struct object_list *not_mapped = NULL; + unsatisfy_all_pseudo_merges(bitmap_git); + + if (bitmap_git->pseudo_merges.nr) { + struct bitmap *roots_bitmap = bitmap_new(); + struct object_list *objects = NULL; + + for (objects = roots; objects; objects = objects->next) { + struct object *object = objects->item; + int pos; + + pos = bitmap_position(bitmap_git, &object->oid); + if (pos < 0) + continue; + + bitmap_set(roots_bitmap, pos); + } + + base = bitmap_new(); + if (!cascade_pseudo_merges_1(bitmap_git, base, roots_bitmap)) + bitmap_free(roots_bitmap); + } + /* * Go through all the roots for the walk. The ones that have bitmaps * on the bitmap index will be `or`ed together to form an initial @@ -1216,11 +1401,21 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git, */ while (roots) { struct object *object = roots->item; + roots = roots->next; + if (base) { + int pos = bitmap_position(bitmap_git, &object->oid); + if (pos > 0 && bitmap_get(base, pos)) { + object->flags |= SEEN; + continue; + } + } + if (object->type == OBJ_COMMIT && add_commit_to_bitmap(bitmap_git, &base, (struct commit *)object)) { object->flags |= SEEN; + existing_bitmaps = 1; continue; } @@ -1236,6 +1431,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git, roots = not_mapped; + if (existing_bitmaps) + cascade_pseudo_merges_1(bitmap_git, base, NULL); + /* * Let's iterate through all the roots that don't have bitmaps to * check if we can determine them to be reachable from the existing @@ -1256,8 +1454,12 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git, object->flags &= ~UNINTERESTING; add_pending_object(revs, object, ""); needs_walk = 1; + + roots_without_bitmaps_nr++; } else { object->flags |= SEEN; + + roots_with_bitmaps_nr++; } } @@ -1820,6 +2022,19 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, object_list_free(&wants); object_list_free(&haves); + trace2_data_intmax("bitmap", the_repository, "pseudo_merges_satisfied", + pseudo_merges_satisfied_nr); + trace2_data_intmax("bitmap", the_repository, "pseudo_merges_cascades", + pseudo_merges_cascades_nr); + trace2_data_intmax("bitmap", the_repository, "bitmap/hits", + existing_bitmaps_hits_nr); + trace2_data_intmax("bitmap", the_repository, "bitmap/misses", + existing_bitmaps_misses_nr); + trace2_data_intmax("bitmap", the_repository, "bitmap/roots_with_bitmap", + roots_with_bitmaps_nr); + trace2_data_intmax("bitmap", the_repository, "bitmap/roots_without_bitmap", + roots_without_bitmaps_nr); + return bitmap_git; cleanup: @@ -2073,6 +2288,7 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, QSORT(packs, packs_nr, bitmapped_pack_cmp); } else { struct packed_git *pack; + uint32_t pack_int_id; if (bitmap_is_midx(bitmap_git)) { uint32_t preferred_pack_pos; @@ -2083,12 +2299,24 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, } pack = bitmap_git->midx->packs[preferred_pack_pos]; + pack_int_id = preferred_pack_pos; } else { pack = bitmap_git->pack; + /* + * Any value for 'pack_int_id' will do here. When we + * process the pack via try_partial_reuse(), we won't + * use the `pack_int_id` field since we have a non-MIDX + * bitmap. + * + * Use '-1' as a sentinel value to make it clear + * that we do not expect to read this field. + */ + pack_int_id = -1; } ALLOC_GROW(packs, packs_nr + 1, packs_alloc); packs[packs_nr].p = pack; + packs[packs_nr].pack_int_id = pack_int_id; packs[packs_nr].bitmap_nr = pack->num_objects; packs[packs_nr].bitmap_pos = 0; @@ -2397,6 +2625,132 @@ cleanup: return 0; } +static void bit_pos_to_object_id(struct bitmap_index *bitmap_git, + uint32_t bit_pos, + struct object_id *oid) +{ + uint32_t index_pos; + + if (bitmap_is_midx(bitmap_git)) + index_pos = pack_pos_to_midx(bitmap_git->midx, bit_pos); + else + index_pos = pack_pos_to_index(bitmap_git->pack, bit_pos); + + nth_bitmap_object_oid(bitmap_git, oid, index_pos); +} + +int test_bitmap_pseudo_merges(struct repository *r) +{ + struct bitmap_index *bitmap_git; + uint32_t i; + + bitmap_git = prepare_bitmap_git(r); + if (!bitmap_git || !bitmap_git->pseudo_merges.nr) + goto cleanup; + + for (i = 0; i < bitmap_git->pseudo_merges.nr; i++) { + struct pseudo_merge *merge; + struct ewah_bitmap *commits_bitmap, *merge_bitmap; + + merge = use_pseudo_merge(&bitmap_git->pseudo_merges, + &bitmap_git->pseudo_merges.v[i]); + commits_bitmap = merge->commits; + merge_bitmap = pseudo_merge_bitmap(&bitmap_git->pseudo_merges, + merge); + + printf("at=%"PRIuMAX", commits=%"PRIuMAX", objects=%"PRIuMAX"\n", + (uintmax_t)merge->at, + (uintmax_t)ewah_bitmap_popcount(commits_bitmap), + (uintmax_t)ewah_bitmap_popcount(merge_bitmap)); + } + +cleanup: + free_bitmap_index(bitmap_git); + return 0; +} + +static void dump_ewah_object_ids(struct bitmap_index *bitmap_git, + struct ewah_bitmap *bitmap) + +{ + struct ewah_iterator it; + eword_t word; + uint32_t pos = 0; + + ewah_iterator_init(&it, bitmap); + + while (ewah_iterator_next(&word, &it)) { + struct object_id oid; + uint32_t offset; + + for (offset = 0; offset < BITS_IN_EWORD; offset++) { + if (!(word >> offset)) + break; + + offset += ewah_bit_ctz64(word >> offset); + + bit_pos_to_object_id(bitmap_git, pos + offset, &oid); + printf("%s\n", oid_to_hex(&oid)); + } + pos += BITS_IN_EWORD; + } +} + +int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n) +{ + struct bitmap_index *bitmap_git; + struct pseudo_merge *merge; + int ret = 0; + + bitmap_git = prepare_bitmap_git(r); + if (!bitmap_git || !bitmap_git->pseudo_merges.nr) + goto cleanup; + + if (n >= bitmap_git->pseudo_merges.nr) { + ret = error(_("pseudo-merge index out of range " + "(%"PRIu32" >= %"PRIuMAX")"), + n, (uintmax_t)bitmap_git->pseudo_merges.nr); + goto cleanup; + } + + merge = use_pseudo_merge(&bitmap_git->pseudo_merges, + &bitmap_git->pseudo_merges.v[n]); + dump_ewah_object_ids(bitmap_git, merge->commits); + +cleanup: + free_bitmap_index(bitmap_git); + return ret; +} + +int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n) +{ + struct bitmap_index *bitmap_git; + struct pseudo_merge *merge; + int ret = 0; + + bitmap_git = prepare_bitmap_git(r); + if (!bitmap_git || !bitmap_git->pseudo_merges.nr) + goto cleanup; + + if (n >= bitmap_git->pseudo_merges.nr) { + ret = error(_("pseudo-merge index out of range " + "(%"PRIu32" >= %"PRIuMAX")"), + n, (uintmax_t)bitmap_git->pseudo_merges.nr); + goto cleanup; + } + + merge = use_pseudo_merge(&bitmap_git->pseudo_merges, + &bitmap_git->pseudo_merges.v[n]); + + dump_ewah_object_ids(bitmap_git, + pseudo_merge_bitmap(&bitmap_git->pseudo_merges, + merge)); + +cleanup: + free_bitmap_index(bitmap_git); + return ret; +} + int rebuild_bitmap(const uint32_t *reposition, struct ewah_bitmap *source, struct bitmap *dest) @@ -2503,6 +2857,7 @@ void free_bitmap_index(struct bitmap_index *b) */ close_midx_revindex(b->midx); } + free_pseudo_merge_map(&b->pseudo_merges); free(b); } diff --git a/pack-bitmap.h b/pack-bitmap.h index 3091095f33..1171e6d989 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -21,6 +21,7 @@ struct bitmap_disk_header { unsigned char checksum[GIT_MAX_RAWSZ]; }; +#define BITMAP_PSEUDO_MERGE (1u<<21) #define NEEDS_BITMAP (1u<<22) /* @@ -36,6 +37,7 @@ enum pack_bitmap_opts { BITMAP_OPT_FULL_DAG = 0x1, BITMAP_OPT_HASH_CACHE = 0x4, BITMAP_OPT_LOOKUP_TABLE = 0x10, + BITMAP_OPT_PSEUDO_MERGES = 0x20, }; enum pack_bitmap_flags { @@ -71,6 +73,9 @@ void traverse_bitmap_commit_list(struct bitmap_index *, void test_bitmap_walk(struct rev_info *revs); int test_bitmap_commits(struct repository *r); int test_bitmap_hashes(struct repository *r); +int test_bitmap_pseudo_merges(struct repository *r); +int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n); +int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n); #define GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL \ "GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL" @@ -109,12 +114,16 @@ struct bitmap_writer { struct bitmapped_commit *selected; unsigned int selected_nr, selected_alloc; + struct string_list pseudo_merge_groups; + kh_oid_map_t *pseudo_merge_commits; /* oid -> pseudo merge(s) */ + uint32_t pseudo_merges_nr; + struct progress *progress; int show_progress; unsigned char pack_checksum[GIT_MAX_RAWSZ]; }; -void bitmap_writer_init(struct bitmap_writer *writer); +void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r); void bitmap_writer_show_progress(struct bitmap_writer *writer, int show); void bitmap_writer_set_checksum(struct bitmap_writer *writer, const unsigned char *sha1); @@ -122,6 +131,10 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer, struct packing_data *to_pack, struct pack_idx_entry **index, uint32_t index_nr); +int bitmap_writer_has_bitmapped_object_id(struct bitmap_writer *writer, + const struct object_id *oid); +void bitmap_writer_push_commit(struct bitmap_writer *writer, + struct commit *commit, unsigned pseudo_merge); uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git, struct packing_data *mapping); int rebuild_bitmap(const uint32_t *reposition, @@ -129,6 +142,8 @@ int rebuild_bitmap(const uint32_t *reposition, struct bitmap *dest); struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git, struct commit *commit); +struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git, + struct commit *commit); void bitmap_writer_select_commits(struct bitmap_writer *writer, struct commit **indexed_commits, unsigned int indexed_commits_nr); @@ -150,4 +165,6 @@ int bitmap_is_preferred_refname(struct repository *r, const char *refname); int verify_bitmap_files(struct repository *r); +struct ewah_bitmap *read_bitmap(const unsigned char *map, + size_t map_size, size_t *map_pos); #endif diff --git a/pack-revindex.c b/pack-revindex.c index fc63aa76a2..93ffca7731 100644 --- a/pack-revindex.c +++ b/pack-revindex.c @@ -527,6 +527,9 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m, { uint32_t *found; + if (key->pack >= m->num_packs) + BUG("MIDX pack lookup out of bounds (%"PRIu32" >= %"PRIu32")", + key->pack, m->num_packs); /* * The preferred pack sorts first, so determine its identifier by * looking at the first object in pseudo-pack order. diff --git a/parse-options-cb.c b/parse-options-cb.c index d99d688d3c..b2aa62a9dc 100644 --- a/parse-options-cb.c +++ b/parse-options-cb.c @@ -30,8 +30,6 @@ int parse_opt_abbrev_cb(const struct option *opt, const char *arg, int unset) opt->long_name); if (v && v < MINIMUM_ABBREV) v = MINIMUM_ABBREV; - else if (startup_info->have_repository && v > the_hash_algo->hexsz) - v = the_hash_algo->hexsz; } *(int *)(opt->value) = v; return 0; @@ -125,6 +125,35 @@ int git_parse_ssize_t(const char *value, ssize_t *ret) return 1; } +int git_parse_double(const char *value, double *ret) +{ + char *end; + double val; + uintmax_t factor; + + if (!value || !*value) { + errno = EINVAL; + return 0; + } + + errno = 0; + val = strtod(value, &end); + if (errno == ERANGE) + return 0; + if (end == value) { + errno = EINVAL; + return 0; + } + factor = get_unit_factor(end); + if (!factor) { + errno = EINVAL; + return 0; + } + val *= factor; + *ret = val; + return 1; +} + int git_parse_maybe_bool_text(const char *value) { if (!value) @@ -6,6 +6,7 @@ int git_parse_ssize_t(const char *, ssize_t *); int git_parse_ulong(const char *, unsigned long *); int git_parse_int(const char *value, int *ret); int git_parse_int64(const char *value, int64_t *ret); +int git_parse_double(const char *value, double *ret); /** * Same as `git_config_bool`, except that it returns -1 on error rather @@ -4,6 +4,7 @@ struct repository; struct strbuf; struct string_list; +struct worktree; /* * The result to all functions which return statically allocated memory may be @@ -82,6 +83,14 @@ const char *git_path(const char *fmt, ...) __attribute__((format (printf, 1, 2))); /* + * Similar to git_path() but can produce paths for a specified + * worktree instead of current one + */ +const char *worktree_git_path(const struct worktree *wt, + const char *fmt, ...) + __attribute__((format (printf, 2, 3))); + +/* * Return a path into the main repository's (the_repository) git directory. */ char *git_pathdup(const char *fmt, ...) diff --git a/pseudo-merge.c b/pseudo-merge.c new file mode 100644 index 0000000000..e3e0393f11 --- /dev/null +++ b/pseudo-merge.c @@ -0,0 +1,757 @@ +#include "git-compat-util.h" +#include "pseudo-merge.h" +#include "date.h" +#include "oid-array.h" +#include "strbuf.h" +#include "config.h" +#include "string-list.h" +#include "refs.h" +#include "pack-bitmap.h" +#include "commit.h" +#include "alloc.h" +#include "progress.h" +#include "hex.h" + +#define DEFAULT_PSEUDO_MERGE_DECAY 1.0 +#define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64 +#define DEFAULT_PSEUDO_MERGE_SAMPLE_RATE 1 +#define DEFAULT_PSEUDO_MERGE_THRESHOLD approxidate("1.week.ago") +#define DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD approxidate("1.month.ago") +#define DEFAULT_PSEUDO_MERGE_STABLE_SIZE 512 + +static double gitexp(double base, int exp) +{ + double result = 1; + while (1) { + if (exp % 2) + result *= base; + exp >>= 1; + if (!exp) + break; + base *= base; + } + return result; +} + +static uint32_t pseudo_merge_group_size(const struct pseudo_merge_group *group, + const struct pseudo_merge_matches *matches, + uint32_t i) +{ + double C = 0.0f; + uint32_t n; + + /* + * The size of pseudo-merge groups decays according to a power series, + * which looks like: + * + * f(n) = C * n^-k + * + * , where 'n' is the n-th pseudo-merge group, 'f(n)' is its size, 'k' + * is the decay rate, and 'C' is a scaling value. + * + * The value of C depends on the number of groups, decay rate, and total + * number of commits. It is computed such that if there are M and N + * total groups and commits, respectively, that: + * + * N = f(0) + f(1) + ... f(M-1) + * + * Rearranging to isolate C, we get: + * + * N = \sum_{n=1}^M C / n^k + * + * N / C = \sum_{n=1}^M n^-k + * + * C = N / \sum_{n=1}^M n^-k + * + * For example, if we have a decay rate of 'k' being equal to 1.5, 'N' + * total commits equal to 10,000, and 'M' being equal to 6 groups, then + * the (rounded) group sizes are: + * + * { 5469, 1934, 1053, 684, 489, 372 } + * + * increasing the number of total groups, say to 10, scales the group + * sizes appropriately: + * + * { 5012, 1772, 964, 626, 448, 341, 271, 221, 186, 158 } + */ + for (n = 0; n < group->max_merges; n++) + C += 1.0 / gitexp(n + 1, group->decay); + C = matches->unstable_nr / C; + + return (uint32_t)((C / gitexp(i + 1, group->decay)) + 0.5); +} + +static void pseudo_merge_group_init(struct pseudo_merge_group *group) +{ + memset(group, 0, sizeof(struct pseudo_merge_group)); + + strmap_init_with_options(&group->matches, NULL, 0); + + group->decay = DEFAULT_PSEUDO_MERGE_DECAY; + group->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES; + group->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE; + group->threshold = DEFAULT_PSEUDO_MERGE_THRESHOLD; + group->stable_threshold = DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD; + group->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE; +} + +static int pseudo_merge_config(const char *var, const char *value, + const struct config_context *ctx, + void *cb_data) +{ + struct string_list *list = cb_data; + struct string_list_item *item; + struct pseudo_merge_group *group; + struct strbuf buf = STRBUF_INIT; + const char *sub, *key; + size_t sub_len; + int ret = 0; + + if (parse_config_key(var, "bitmappseudomerge", &sub, &sub_len, &key)) + goto done; + + if (!sub_len) + goto done; + + strbuf_add(&buf, sub, sub_len); + + item = string_list_lookup(list, buf.buf); + if (!item) { + item = string_list_insert(list, buf.buf); + + item->util = xmalloc(sizeof(struct pseudo_merge_group)); + pseudo_merge_group_init(item->util); + } + + group = item->util; + + if (!strcmp(key, "pattern")) { + struct strbuf re = STRBUF_INIT; + + free(group->pattern); + if (*value != '^') + strbuf_addch(&re, '^'); + strbuf_addstr(&re, value); + + group->pattern = xcalloc(1, sizeof(regex_t)); + if (regcomp(group->pattern, re.buf, REG_EXTENDED)) + die(_("failed to load pseudo-merge regex for %s: '%s'"), + sub, re.buf); + + strbuf_release(&re); + } else if (!strcmp(key, "decay")) { + group->decay = git_config_double(var, value, ctx->kvi); + if (group->decay < 0) { + warning(_("%s must be non-negative, using default"), var); + group->decay = DEFAULT_PSEUDO_MERGE_DECAY; + } + } else if (!strcmp(key, "samplerate")) { + group->sample_rate = git_config_double(var, value, ctx->kvi); + if (!(0 <= group->sample_rate && group->sample_rate <= 1)) { + warning(_("%s must be between 0 and 1, using default"), var); + group->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE; + } + } else if (!strcmp(key, "threshold")) { + if (git_config_expiry_date(&group->threshold, var, value)) { + ret = -1; + goto done; + } + } else if (!strcmp(key, "maxmerges")) { + group->max_merges = git_config_int(var, value, ctx->kvi); + if (group->max_merges < 0) { + warning(_("%s must be non-negative, using default"), var); + group->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES; + } + } else if (!strcmp(key, "stablethreshold")) { + if (git_config_expiry_date(&group->stable_threshold, var, value)) { + ret = -1; + goto done; + } + } else if (!strcmp(key, "stablesize")) { + group->stable_size = git_config_int(var, value, ctx->kvi); + if (group->stable_size <= 0) { + warning(_("%s must be positive, using default"), var); + group->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE; + } + } + +done: + strbuf_release(&buf); + + return ret; +} + +void load_pseudo_merges_from_config(struct string_list *list) +{ + struct string_list_item *item; + + git_config(pseudo_merge_config, list); + + for_each_string_list_item(item, list) { + struct pseudo_merge_group *group = item->util; + if (!group->pattern) + die(_("pseudo-merge group '%s' missing required pattern"), + item->string); + if (group->threshold < group->stable_threshold) + die(_("pseudo-merge group '%s' has unstable threshold " + "before stable one"), item->string); + } +} + +static int find_pseudo_merge_group_for_ref(const char *refname, + const struct object_id *oid, + int flags UNUSED, + void *_data) +{ + struct bitmap_writer *writer = _data; + struct object_id peeled; + struct commit *c; + uint32_t i; + int has_bitmap; + + if (!peel_iterated_oid(the_repository, oid, &peeled)) + oid = &peeled; + + c = lookup_commit(the_repository, oid); + if (!c) + return 0; + + has_bitmap = bitmap_writer_has_bitmapped_object_id(writer, oid); + + for (i = 0; i < writer->pseudo_merge_groups.nr; i++) { + struct pseudo_merge_group *group; + struct pseudo_merge_matches *matches; + struct strbuf group_name = STRBUF_INIT; + regmatch_t captures[16]; + size_t j; + + group = writer->pseudo_merge_groups.items[i].util; + if (regexec(group->pattern, refname, ARRAY_SIZE(captures), + captures, 0)) + continue; + + if (captures[ARRAY_SIZE(captures) - 1].rm_so != -1) + warning(_("pseudo-merge regex from config has too many capture " + "groups (max=%"PRIuMAX")"), + (uintmax_t)ARRAY_SIZE(captures) - 2); + + for (j = !!group->pattern->re_nsub; j < ARRAY_SIZE(captures); j++) { + regmatch_t *match = &captures[j]; + if (match->rm_so == -1) + continue; + + if (group_name.len) + strbuf_addch(&group_name, '-'); + + strbuf_add(&group_name, refname + match->rm_so, + match->rm_eo - match->rm_so); + } + + matches = strmap_get(&group->matches, group_name.buf); + if (!matches) { + matches = xcalloc(1, sizeof(*matches)); + strmap_put(&group->matches, strbuf_detach(&group_name, NULL), + matches); + } + + if (c->date <= group->stable_threshold) { + ALLOC_GROW(matches->stable, matches->stable_nr + 1, + matches->stable_alloc); + matches->stable[matches->stable_nr++] = c; + } else if (c->date <= group->threshold && !has_bitmap) { + ALLOC_GROW(matches->unstable, matches->unstable_nr + 1, + matches->unstable_alloc); + matches->unstable[matches->unstable_nr++] = c; + } + + strbuf_release(&group_name); + } + + return 0; +} + +static struct commit *push_pseudo_merge(struct pseudo_merge_group *group) +{ + struct commit *merge; + + ALLOC_GROW(group->merges, group->merges_nr + 1, group->merges_alloc); + + merge = alloc_commit_node(the_repository); + merge->object.parsed = 1; + merge->object.flags |= BITMAP_PSEUDO_MERGE; + + group->merges[group->merges_nr++] = merge; + + return merge; +} + +static struct pseudo_merge_commit_idx *pseudo_merge_idx(kh_oid_map_t *pseudo_merge_commits, + const struct object_id *oid) + +{ + struct pseudo_merge_commit_idx *pmc; + int hash_ret; + khiter_t hash_pos = kh_put_oid_map(pseudo_merge_commits, *oid, + &hash_ret); + + if (hash_ret) { + CALLOC_ARRAY(pmc, 1); + kh_value(pseudo_merge_commits, hash_pos) = pmc; + } else { + pmc = kh_value(pseudo_merge_commits, hash_pos); + } + + return pmc; +} + +#define MIN_PSEUDO_MERGE_SIZE 8 + +static void select_pseudo_merges_1(struct bitmap_writer *writer, + struct pseudo_merge_group *group, + struct pseudo_merge_matches *matches) +{ + uint32_t i, j; + uint32_t stable_merges_nr; + + if (!matches->stable_nr && !matches->unstable_nr) + return; /* all tips in this group already have bitmaps */ + + stable_merges_nr = matches->stable_nr / group->stable_size; + if (matches->stable_nr % group->stable_size) + stable_merges_nr++; + + /* make stable_merges_nr pseudo merges for stable commits */ + for (i = 0, j = 0; i < stable_merges_nr; i++) { + struct commit *merge; + struct commit_list **p; + + merge = push_pseudo_merge(group); + p = &merge->parents; + + /* + * For each pseudo-merge created above, add parents to the + * allocated commit node from the stable set of commits + * (un-bitmapped, newer than the stable threshold). + */ + do { + struct commit *c; + struct pseudo_merge_commit_idx *pmc; + + if (j >= matches->stable_nr) + break; + + c = matches->stable[j++]; + /* + * Here and below, make sure that we keep our mapping of + * commits -> pseudo-merge(s) which include the key'd + * commit up-to-date. + */ + pmc = pseudo_merge_idx(writer->pseudo_merge_commits, + &c->object.oid); + + ALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc); + + pmc->pseudo_merge[pmc->nr++] = writer->pseudo_merges_nr; + p = commit_list_append(c, p); + } while (j % group->stable_size); + + bitmap_writer_push_commit(writer, merge, 1); + writer->pseudo_merges_nr++; + } + + /* make up to group->max_merges pseudo merges for unstable commits */ + for (i = 0, j = 0; i < group->max_merges; i++) { + struct commit *merge; + struct commit_list **p; + uint32_t size, end; + + merge = push_pseudo_merge(group); + p = &merge->parents; + + size = pseudo_merge_group_size(group, matches, i); + end = size < MIN_PSEUDO_MERGE_SIZE ? matches->unstable_nr : j + size; + + /* + * For each pseudo-merge commit created above, add parents to + * the allocated commit node from the unstable set of commits + * (newer than the stable threshold). + * + * Account for the sample rate, since not every candidate from + * the set of stable commits will be included as a pseudo-merge + * parent. + */ + for (; j < end && j < matches->unstable_nr; j++) { + struct commit *c = matches->unstable[j]; + struct pseudo_merge_commit_idx *pmc; + + if (j % (uint32_t)(1.0 / group->sample_rate)) + continue; + + pmc = pseudo_merge_idx(writer->pseudo_merge_commits, + &c->object.oid); + + ALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc); + + pmc->pseudo_merge[pmc->nr++] = writer->pseudo_merges_nr; + p = commit_list_append(c, p); + } + + bitmap_writer_push_commit(writer, merge, 1); + writer->pseudo_merges_nr++; + if (end >= matches->unstable_nr) + break; + } +} + +static int commit_date_cmp(const void *va, const void *vb) +{ + timestamp_t a = (*(const struct commit **)va)->date; + timestamp_t b = (*(const struct commit **)vb)->date; + + if (a < b) + return -1; + else if (a > b) + return 1; + return 0; +} + +static void sort_pseudo_merge_matches(struct pseudo_merge_matches *matches) +{ + QSORT(matches->stable, matches->stable_nr, commit_date_cmp); + QSORT(matches->unstable, matches->unstable_nr, commit_date_cmp); +} + +void select_pseudo_merges(struct bitmap_writer *writer, + struct commit **commits, size_t commits_nr) +{ + struct progress *progress = NULL; + uint32_t i; + + if (!writer->pseudo_merge_groups.nr) + return; + + if (writer->show_progress) + progress = start_progress("Selecting pseudo-merge commits", + writer->pseudo_merge_groups.nr); + + refs_for_each_ref(get_main_ref_store(the_repository), + find_pseudo_merge_group_for_ref, writer); + + for (i = 0; i < writer->pseudo_merge_groups.nr; i++) { + struct pseudo_merge_group *group; + struct hashmap_iter iter; + struct strmap_entry *e; + + group = writer->pseudo_merge_groups.items[i].util; + strmap_for_each_entry(&group->matches, &iter, e) { + struct pseudo_merge_matches *matches = e->value; + + sort_pseudo_merge_matches(matches); + + select_pseudo_merges_1(writer, group, matches); + } + + display_progress(progress, i + 1); + } + + stop_progress(&progress); +} + +void free_pseudo_merge_map(struct pseudo_merge_map *pm) +{ + uint32_t i; + for (i = 0; i < pm->nr; i++) { + ewah_pool_free(pm->v[i].commits); + ewah_pool_free(pm->v[i].bitmap); + } + free(pm->v); +} + +struct pseudo_merge_commit_ext { + uint32_t nr; + const unsigned char *ptr; +}; + +static int pseudo_merge_ext_at(const struct pseudo_merge_map *pm, + struct pseudo_merge_commit_ext *ext, size_t at) +{ + if (at >= pm->map_size) + return error(_("extended pseudo-merge read out-of-bounds " + "(%"PRIuMAX" >= %"PRIuMAX")"), + (uintmax_t)at, (uintmax_t)pm->map_size); + if (at + 4 >= pm->map_size) + return error(_("extended pseudo-merge entry is too short " + "(%"PRIuMAX" >= %"PRIuMAX")"), + (uintmax_t)(at + 4), (uintmax_t)pm->map_size); + + ext->nr = get_be32(pm->map + at); + ext->ptr = pm->map + at + sizeof(uint32_t); + + return 0; +} + +struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm, + struct pseudo_merge *merge) +{ + if (!merge->loaded_commits) + BUG("cannot use unloaded pseudo-merge bitmap"); + + if (!merge->loaded_bitmap) { + size_t at = merge->bitmap_at; + + merge->bitmap = read_bitmap(pm->map, pm->map_size, &at); + merge->loaded_bitmap = 1; + } + + return merge->bitmap; +} + +struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm, + struct pseudo_merge *merge) +{ + if (!merge->loaded_commits) { + size_t pos = merge->at; + + merge->commits = read_bitmap(pm->map, pm->map_size, &pos); + merge->bitmap_at = pos; + merge->loaded_commits = 1; + } + return merge; +} + +static struct pseudo_merge *pseudo_merge_at(const struct pseudo_merge_map *pm, + struct object_id *oid, + size_t want) +{ + size_t lo = 0; + size_t hi = pm->nr; + + while (lo < hi) { + size_t mi = lo + (hi - lo) / 2; + size_t got = pm->v[mi].at; + + if (got == want) + return use_pseudo_merge(pm, &pm->v[mi]); + else if (got < want) + hi = mi; + else + lo = mi + 1; + } + + warning(_("could not find pseudo-merge for commit %s at offset %"PRIuMAX), + oid_to_hex(oid), (uintmax_t)want); + + return NULL; +} + +struct pseudo_merge_commit { + uint32_t commit_pos; + uint64_t pseudo_merge_ofs; +}; + +#define PSEUDO_MERGE_COMMIT_RAWSZ (sizeof(uint32_t)+sizeof(uint64_t)) + +static void read_pseudo_merge_commit_at(struct pseudo_merge_commit *merge, + const unsigned char *at) +{ + merge->commit_pos = get_be32(at); + merge->pseudo_merge_ofs = get_be64(at + sizeof(uint32_t)); +} + +static int nth_pseudo_merge_ext(const struct pseudo_merge_map *pm, + struct pseudo_merge_commit_ext *ext, + struct pseudo_merge_commit *merge, + uint32_t n) +{ + size_t ofs; + + if (n >= ext->nr) + return error(_("extended pseudo-merge lookup out-of-bounds " + "(%"PRIu32" >= %"PRIu32")"), n, ext->nr); + + ofs = get_be64(ext->ptr + st_mult(n, sizeof(uint64_t))); + if (ofs >= pm->map_size) + return error(_("out-of-bounds read: (%"PRIuMAX" >= %"PRIuMAX")"), + (uintmax_t)ofs, (uintmax_t)pm->map_size); + + read_pseudo_merge_commit_at(merge, pm->map + ofs); + + return 0; +} + +static unsigned apply_pseudo_merge(const struct pseudo_merge_map *pm, + struct pseudo_merge *merge, + struct bitmap *result, + struct bitmap *roots) +{ + if (merge->satisfied) + return 0; + + if (!ewah_bitmap_is_subset(merge->commits, roots ? roots : result)) + return 0; + + bitmap_or_ewah(result, pseudo_merge_bitmap(pm, merge)); + if (roots) + bitmap_or_ewah(roots, pseudo_merge_bitmap(pm, merge)); + merge->satisfied = 1; + + return 1; +} + +static int pseudo_merge_commit_cmp(const void *va, const void *vb) +{ + struct pseudo_merge_commit merge; + uint32_t key = *(uint32_t*)va; + + read_pseudo_merge_commit_at(&merge, vb); + + if (key < merge.commit_pos) + return -1; + if (key > merge.commit_pos) + return 1; + return 0; +} + +static struct pseudo_merge_commit *find_pseudo_merge(const struct pseudo_merge_map *pm, + uint32_t pos) +{ + if (!pm->commits_nr) + return NULL; + + return bsearch(&pos, pm->commits, pm->commits_nr, + PSEUDO_MERGE_COMMIT_RAWSZ, pseudo_merge_commit_cmp); +} + +int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm, + struct bitmap *result, + struct commit *commit, uint32_t commit_pos) +{ + struct pseudo_merge *merge; + struct pseudo_merge_commit *merge_commit; + int ret = 0; + + merge_commit = find_pseudo_merge(pm, commit_pos); + if (!merge_commit) + return 0; + + if (merge_commit->pseudo_merge_ofs & ((uint64_t)1<<63)) { + struct pseudo_merge_commit_ext ext = { 0 }; + off_t ofs = merge_commit->pseudo_merge_ofs & ~((uint64_t)1<<63); + uint32_t i; + + if (pseudo_merge_ext_at(pm, &ext, ofs) < -1) { + warning(_("could not read extended pseudo-merge table " + "for commit %s"), + oid_to_hex(&commit->object.oid)); + return ret; + } + + for (i = 0; i < ext.nr; i++) { + if (nth_pseudo_merge_ext(pm, &ext, merge_commit, i) < 0) + return ret; + + merge = pseudo_merge_at(pm, &commit->object.oid, + merge_commit->pseudo_merge_ofs); + + if (!merge) + return ret; + + if (apply_pseudo_merge(pm, merge, result, NULL)) + ret++; + } + } else { + merge = pseudo_merge_at(pm, &commit->object.oid, + merge_commit->pseudo_merge_ofs); + + if (!merge) + return ret; + + if (apply_pseudo_merge(pm, merge, result, NULL)) + ret++; + } + + if (ret) + cascade_pseudo_merges(pm, result, NULL); + + return ret; +} + +int cascade_pseudo_merges(const struct pseudo_merge_map *pm, + struct bitmap *result, + struct bitmap *roots) +{ + unsigned any_satisfied; + int ret = 0; + + do { + struct pseudo_merge *merge; + uint32_t i; + + any_satisfied = 0; + + for (i = 0; i < pm->nr; i++) { + merge = use_pseudo_merge(pm, &pm->v[i]); + if (apply_pseudo_merge(pm, merge, result, roots)) { + any_satisfied |= 1; + ret++; + } + } + } while (any_satisfied); + + return ret; +} + +struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm, + struct bitmap *parents) +{ + struct pseudo_merge *match = NULL; + size_t i; + + if (!pm->nr) + return NULL; + + /* + * NOTE: this loop is quadratic in the worst-case (where no + * matching pseudo-merge bitmaps are found), but in practice + * this is OK for a few reasons: + * + * - Rejecting pseudo-merge bitmaps that do not match the + * given commit is done quickly (i.e. `bitmap_equals_ewah()` + * returns early when we know the two bitmaps aren't equal. + * + * - Already matched pseudo-merge bitmaps (which we track with + * the `->satisfied` bit here) are skipped as potential + * candidates. + * + * - The number of pseudo-merges should be small (in the + * hundreds for most repositories). + * + * If in the future this semi-quadratic behavior does become a + * problem, another approach would be to keep track of which + * pseudo-merges are still "viable" after enumerating the + * pseudo-merge commit's parents: + * + * - A pseudo-merge bitmap becomes non-viable when the bit(s) + * corresponding to one or more parent(s) of the given + * commit are not set in a candidate pseudo-merge's commits + * bitmap. + * + * - After processing all bits, enumerate the remaining set of + * viable pseudo-merge bitmaps, and check that their + * popcount() matches the number of parents in the given + * commit. + */ + for (i = 0; i < pm->nr; i++) { + struct pseudo_merge *candidate = use_pseudo_merge(pm, &pm->v[i]); + if (!candidate || candidate->satisfied) + continue; + if (!bitmap_equals_ewah(parents, candidate->commits)) + continue; + + match = candidate; + match->satisfied = 1; + break; + } + + return match; +} diff --git a/pseudo-merge.h b/pseudo-merge.h new file mode 100644 index 0000000000..2aca01d056 --- /dev/null +++ b/pseudo-merge.h @@ -0,0 +1,216 @@ +#ifndef PSEUDO_MERGE_H +#define PSEUDO_MERGE_H + +#include "git-compat-util.h" +#include "strmap.h" +#include "khash.h" +#include "ewah/ewok.h" + +struct commit; +struct string_list; +struct bitmap_index; +struct bitmap_writer; + +/* + * A pseudo-merge group tracks the set of non-bitmapped reference tips + * that match the given pattern. + * + * Within those matches, they are further segmented by separating + * consecutive capture groups with '-' dash character capture groups + * with '-' dash characters. + * + * Those groups are then ordered by committer date and partitioned + * into individual pseudo-merge(s) according to the decay, max_merges, + * sample_rate, and threshold parameters. + */ +struct pseudo_merge_group { + regex_t *pattern; + + /* capture group(s) -> struct pseudo_merge_matches */ + struct strmap matches; + + /* + * The individual pseudo-merge(s) that are generated from the + * above array of matches, partitioned according to the below + * parameters. + */ + struct commit **merges; + size_t merges_nr; + size_t merges_alloc; + + /* + * Pseudo-merge grouping parameters. See git-config(1) for + * more information. + */ + double decay; + int max_merges; + double sample_rate; + int stable_size; + timestamp_t threshold; + timestamp_t stable_threshold; +}; + +struct pseudo_merge_matches { + struct commit **stable; + struct commit **unstable; + size_t stable_nr, stable_alloc; + size_t unstable_nr, unstable_alloc; +}; + +/* + * Read the repository's configuration: + * + * - bitmapPseudoMerge.<name>.pattern + * - bitmapPseudoMerge.<name>.decay + * - bitmapPseudoMerge.<name>.sampleRate + * - bitmapPseudoMerge.<name>.threshold + * - bitmapPseudoMerge.<name>.maxMerges + * - bitmapPseudoMerge.<name>.stableThreshold + * - bitmapPseudoMerge.<name>.stableSize + * + * and populates the given `list` with pseudo-merge groups. String + * entry keys are the pseudo-merge group names, and the values are + * pointers to the pseudo_merge_group structure itself. + */ +void load_pseudo_merges_from_config(struct string_list *list); + +/* + * A pseudo-merge commit index (pseudo_merge_commit_idx) maps a + * particular (non-pseudo-merge) commit to the list of pseudo-merge(s) + * it appears in. + */ +struct pseudo_merge_commit_idx { + uint32_t *pseudo_merge; + size_t nr, alloc; +}; + +/* + * Selects pseudo-merges from a list of commits, populating the given + * string_list of pseudo-merge groups. + * + * Populates the pseudo_merge_commits map with a commit_idx + * corresponding to each commit in the list. Counts the total number + * of pseudo-merges generated. + * + * Optionally shows a progress meter. + */ +void select_pseudo_merges(struct bitmap_writer *writer, + struct commit **commits, size_t commits_nr); + +/* + * Represents a serialized view of a file containing pseudo-merge(s) + * (see Documentation/technical/bitmap-format.txt for a specification + * of the format). + */ +struct pseudo_merge_map { + /* + * An array of pseudo-merge(s), lazily loaded from the .bitmap + * file. + */ + struct pseudo_merge *v; + size_t nr; + size_t commits_nr; + + /* + * Pointers into a memory-mapped view of the .bitmap file: + * + * - map: the beginning of the .bitmap file + * - commits: the beginning of the pseudo-merge commit index + * - map_size: the size of the .bitmap file + */ + const unsigned char *map; + const unsigned char *commits; + + size_t map_size; +}; + +/* + * An individual pseudo-merge, storing a pair of lazily-loaded + * bitmaps: + * + * - commits: the set of commit(s) that are part of the pseudo-merge + * - bitmap: the set of object(s) reachable from the above set of + * commits. + * + * The `at` and `bitmap_at` fields are used to store the locations of + * each of the above bitmaps in the .bitmap file. + */ +struct pseudo_merge { + struct ewah_bitmap *commits; + struct ewah_bitmap *bitmap; + + off_t at; + off_t bitmap_at; + + /* + * `satisfied` indicates whether the given pseudo-merge has been + * used. + * + * `loaded_commits` and `loaded_bitmap` indicate whether the + * respective bitmaps have been loaded and read from the + * .bitmap file. + */ + unsigned satisfied : 1, + loaded_commits : 1, + loaded_bitmap : 1; +}; + +/* + * Frees the given pseudo-merge map, releasing any memory held by (a) + * parsed EWAH bitmaps, or (b) the array of pseudo-merges itself. Does + * not free the memory-mapped view of the .bitmap file. + */ +void free_pseudo_merge_map(struct pseudo_merge_map *pm); + +/* + * Loads the bitmap corresponding to the given pseudo-merge from the + * map, if it has not already been loaded. + */ +struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm, + struct pseudo_merge *merge); + +/* + * Loads the pseudo-merge and its commits bitmap from the given + * pseudo-merge map, if it has not already been loaded. + */ +struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm, + struct pseudo_merge *merge); + +/* + * Applies pseudo-merge(s) containing the given commit to the bitmap + * "result". + * + * If any pseudo-merge(s) were satisfied, returns the number + * satisfied, otherwise returns 0. If any were satisfied, the + * remaining unsatisfied pseudo-merges are cascaded (see below). + */ +int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm, + struct bitmap *result, + struct commit *commit, uint32_t commit_pos); + +/* + * Applies pseudo-merge(s) which are satisfied according to the + * current bitmap in result (or roots, see below). If any + * pseudo-merges were satisfied, repeat the process over unsatisfied + * pseudo-merge commits until no more pseudo-merges are satisfied. + * + * Result is the bitmap to which the pseudo-merge(s) are applied. + * Roots (if given) is a bitmap of the traversal tip(s) for either + * side of a reachability traversal. + * + * Roots may given instead of a populated results bitmap at the + * beginning of a traversal on either side where the reachability + * closure over tips is not yet known. + */ +int cascade_pseudo_merges(const struct pseudo_merge_map *pm, + struct bitmap *result, + struct bitmap *roots); + +/* + * Returns a pseudo-merge which contains the exact set of commits + * listed in the "parents" bitamp, or NULL if none could be found. + */ +struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm, + struct bitmap *parents); + +#endif diff --git a/rebase-interactive.c b/rebase-interactive.c index c343e16fcd..56fd7206a9 100644 --- a/rebase-interactive.c +++ b/rebase-interactive.c @@ -101,9 +101,10 @@ void append_todo_help(int command_count, strbuf_add_commented_lines(buf, msg, strlen(msg), comment_line_str); } -int edit_todo_list(struct repository *r, struct todo_list *todo_list, - struct todo_list *new_todo, const char *shortrevisions, - const char *shortonto, unsigned flags) +int edit_todo_list(struct repository *r, struct replay_opts *opts, + struct todo_list *todo_list, struct todo_list *new_todo, + const char *shortrevisions, const char *shortonto, + unsigned flags) { const char *todo_file = rebase_path_todo(), *todo_backup = rebase_path_todo_backup(); @@ -114,7 +115,9 @@ int edit_todo_list(struct repository *r, struct todo_list *todo_list, * it. If there is an error, we do not return, because the user * might want to fix it in the first place. */ if (!initial) - incorrect = todo_list_parse_insn_buffer(r, todo_list->buf.buf, todo_list) | + incorrect = todo_list_parse_insn_buffer(r, opts, + todo_list->buf.buf, + todo_list) | file_exists(rebase_path_dropped()); if (todo_list_write_to_file(r, todo_list, todo_file, shortrevisions, shortonto, @@ -134,13 +137,13 @@ int edit_todo_list(struct repository *r, struct todo_list *todo_list, if (initial && new_todo->buf.len == 0) return -3; - if (todo_list_parse_insn_buffer(r, new_todo->buf.buf, new_todo)) { + if (todo_list_parse_insn_buffer(r, opts, new_todo->buf.buf, new_todo)) { fprintf(stderr, _(edit_todo_list_advice)); return -4; } if (incorrect) { - if (todo_list_check_against_backup(r, new_todo)) { + if (todo_list_check_against_backup(r, opts, new_todo)) { write_file(rebase_path_dropped(), "%s", ""); return -4; } @@ -228,13 +231,15 @@ leave_check: return res; } -int todo_list_check_against_backup(struct repository *r, struct todo_list *todo_list) +int todo_list_check_against_backup(struct repository *r, + struct replay_opts *opts, + struct todo_list *todo_list) { struct todo_list backup = TODO_LIST_INIT; int res = 0; if (strbuf_read_file(&backup.buf, rebase_path_todo_backup(), 0) > 0) { - todo_list_parse_insn_buffer(r, backup.buf.buf, &backup); + todo_list_parse_insn_buffer(r, opts, backup.buf.buf, &backup); res = todo_list_check(&backup, todo_list); } diff --git a/rebase-interactive.h b/rebase-interactive.h index 7239c60f79..8e5b181b33 100644 --- a/rebase-interactive.h +++ b/rebase-interactive.h @@ -3,17 +3,20 @@ struct strbuf; struct repository; +struct replay_opts; struct todo_list; void append_todo_help(int command_count, const char *shortrevisions, const char *shortonto, struct strbuf *buf); -int edit_todo_list(struct repository *r, struct todo_list *todo_list, - struct todo_list *new_todo, const char *shortrevisions, - const char *shortonto, unsigned flags); +int edit_todo_list(struct repository *r, struct replay_opts *opts, + struct todo_list *todo_list, struct todo_list *new_todo, + const char *shortrevisions, const char *shortonto, + unsigned flags); int todo_list_check(struct todo_list *old_todo, struct todo_list *new_todo); int todo_list_check_against_backup(struct repository *r, + struct replay_opts *opts, struct todo_list *todo_list); #endif @@ -914,7 +914,7 @@ int refs_delete_ref(struct ref_store *refs, const char *msg, transaction = ref_store_transaction_begin(refs, &err); if (!transaction || ref_transaction_delete(transaction, refname, old_oid, - flags, msg, &err) || + NULL, flags, msg, &err) || ref_transaction_commit(transaction, &err)) { error("%s", err.buf); ref_transaction_free(transaction); @@ -1238,43 +1238,57 @@ int ref_transaction_update(struct ref_transaction *transaction, int ref_transaction_create(struct ref_transaction *transaction, const char *refname, const struct object_id *new_oid, + const char *new_target, unsigned int flags, const char *msg, struct strbuf *err) { - if (!new_oid || is_null_oid(new_oid)) { - strbuf_addf(err, "'%s' has a null OID", refname); + if (new_oid && new_target) + BUG("create called with both new_oid and new_target set"); + if ((!new_oid || is_null_oid(new_oid)) && !new_target) { + strbuf_addf(err, "'%s' has neither a valid OID nor a target", refname); return 1; } return ref_transaction_update(transaction, refname, new_oid, - null_oid(), NULL, NULL, flags, + null_oid(), new_target, NULL, flags, msg, err); } int ref_transaction_delete(struct ref_transaction *transaction, const char *refname, const struct object_id *old_oid, - unsigned int flags, const char *msg, + const char *old_target, + unsigned int flags, + const char *msg, struct strbuf *err) { if (old_oid && is_null_oid(old_oid)) BUG("delete called with old_oid set to zeros"); + if (old_oid && old_target) + BUG("delete called with both old_oid and old_target set"); + if (old_target && !(flags & REF_NO_DEREF)) + BUG("delete cannot operate on symrefs with deref mode"); return ref_transaction_update(transaction, refname, null_oid(), old_oid, - NULL, NULL, flags, + NULL, old_target, flags, msg, err); } int ref_transaction_verify(struct ref_transaction *transaction, const char *refname, const struct object_id *old_oid, + const char *old_target, unsigned int flags, struct strbuf *err) { - if (!old_oid) - BUG("verify called with old_oid set to NULL"); + if (!old_target && !old_oid) + BUG("verify called with old_oid and old_target set to NULL"); + if (old_oid && old_target) + BUG("verify called with both old_oid and old_target set"); + if (old_target && !(flags & REF_NO_DEREF)) + BUG("verify cannot operate on symrefs with deref mode"); return ref_transaction_update(transaction, refname, NULL, old_oid, - NULL, NULL, + NULL, old_target, flags, NULL, err); } @@ -2485,7 +2499,7 @@ int refs_delete_refs(struct ref_store *refs, const char *logmsg, for_each_string_list_item(item, refnames) { ret = ref_transaction_delete(transaction, item->string, - NULL, flags, msg, &err); + NULL, NULL, flags, msg, &err); if (ret) { warning(_("could not delete reference %s: %s"), item->string, err.buf); @@ -2595,7 +2609,7 @@ static int migrate_one_ref(const char *refname, const struct object_id *oid, if (ret < 0) goto done; } else { - ret = ref_transaction_create(data->transaction, refname, oid, + ret = ref_transaction_create(data->transaction, refname, oid, NULL, REF_SKIP_CREATE_REFLOG | REF_SKIP_OID_VERIFICATION, NULL, data->errbuf); if (ret < 0) @@ -2878,3 +2892,10 @@ done: strbuf_release(&new_gitdir); return ret; } + +int ref_update_expects_existing_old_ref(struct ref_update *update) +{ + return (update->flags & REF_HAVE_OLD) && + (!is_null_oid(&update->old_oid) || update->old_target); +} + @@ -717,6 +717,7 @@ int ref_transaction_update(struct ref_transaction *transaction, int ref_transaction_create(struct ref_transaction *transaction, const char *refname, const struct object_id *new_oid, + const char *new_target, unsigned int flags, const char *msg, struct strbuf *err); @@ -731,7 +732,9 @@ int ref_transaction_create(struct ref_transaction *transaction, int ref_transaction_delete(struct ref_transaction *transaction, const char *refname, const struct object_id *old_oid, - unsigned int flags, const char *msg, + const char *old_target, + unsigned int flags, + const char *msg, struct strbuf *err); /* @@ -745,6 +748,7 @@ int ref_transaction_delete(struct ref_transaction *transaction, int ref_transaction_verify(struct ref_transaction *transaction, const char *refname, const struct object_id *old_oid, + const char *old_target, unsigned int flags, struct strbuf *err); diff --git a/refs/files-backend.c b/refs/files-backend.c index 4519b46171..44c5c3b201 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -2457,8 +2457,7 @@ static int lock_ref_for_update(struct files_ref_store *refs, struct strbuf *err) { struct strbuf referent = STRBUF_INIT; - int mustexist = (update->flags & REF_HAVE_OLD) && - !is_null_oid(&update->old_oid); + int mustexist = ref_update_expects_existing_old_ref(update); int ret = 0; struct ref_lock *lock; @@ -2537,14 +2536,16 @@ static int lock_ref_for_update(struct files_ref_store *refs, /* * Even if the ref is a regular ref, if `old_target` is set, we - * check the referent value. Ideally `old_target` should only - * be set for symrefs, but we're strict about its usage. + * fail with an error. */ if (update->old_target) { - if (ref_update_check_old_target(referent.buf, update, err)) { - ret = TRANSACTION_GENERIC_ERROR; - goto out; - } + strbuf_addf(err, _("cannot lock ref '%s': " + "expected symref with target '%s': " + "but is a regular ref"), + ref_update_original_update_refname(update), + update->old_target); + ret = TRANSACTION_GENERIC_ERROR; + goto out; } else if (check_old_oid(update, &lock->old_oid, err)) { ret = TRANSACTION_GENERIC_ERROR; goto out; diff --git a/refs/refs-internal.h b/refs/refs-internal.h index cbcb6f9c36..fa975d69aa 100644 --- a/refs/refs-internal.h +++ b/refs/refs-internal.h @@ -742,4 +742,10 @@ int ref_update_has_null_new_value(struct ref_update *update); int ref_update_check_old_target(const char *referent, struct ref_update *update, struct strbuf *err); +/* + * Check if the ref must exist, this means that the old_oid or + * old_target is non NULL. + */ +int ref_update_expects_existing_old_ref(struct ref_update *update); + #endif /* REFS_REFS_INTERNAL_H */ diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 3a75d63eb5..7d872a32ac 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -934,7 +934,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, ¤t_oid, &referent, &u->type); if (ret < 0) goto done; - if (ret > 0 && (!(u->flags & REF_HAVE_OLD) || is_null_oid(&u->old_oid))) { + if (ret > 0 && !ref_update_expects_existing_old_ref(u)) { /* * The reference does not exist, and we either have no * old object ID or expect the reference to not exist. @@ -1005,8 +1005,9 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, */ new_update = ref_transaction_add_update( transaction, referent.buf, new_flags, - &u->new_oid, &u->old_oid, u->new_target, - u->old_target, u->msg); + u->new_target ? NULL : &u->new_oid, + u->old_target ? NULL : &u->old_oid, + u->new_target, u->old_target, u->msg); new_update->parent_update = u; @@ -1038,6 +1039,16 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, * backend returns, which keeps our tests happy. */ if (u->old_target) { + if (!(u->type & REF_ISSYMREF)) { + strbuf_addf(err, _("cannot lock ref '%s': " + "expected symref with target '%s': " + "but is a regular ref"), + ref_update_original_update_refname(u), + u->old_target); + ret = -1; + goto done; + } + if (ref_update_check_old_target(referent.buf, u, err)) { ret = -1; goto done; @@ -200,7 +200,7 @@ struct ref { }; #define REF_NORMAL (1u << 0) -#define REF_HEADS (1u << 1) +#define REF_BRANCHES (1u << 1) #define REF_TAGS (1u << 2) struct ref *find_ref_by_name(const struct ref *list, const char *name); diff --git a/revision.c b/revision.c index 2ee6886078..c68deb57ce 100644 --- a/revision.c +++ b/revision.c @@ -29,6 +29,7 @@ #include "bisect.h" #include "packfile.h" #include "worktree.h" +#include "path.h" #include "read-cache.h" #include "setup.h" #include "sparse-index.h" diff --git a/sequencer.c b/sequencer.c index 30513e87bf..c6a5ccf6dd 100644 --- a/sequencer.c +++ b/sequencer.c @@ -2626,8 +2626,63 @@ static int check_label_or_ref_arg(enum todo_command command, const char *arg) return 0; } -static int parse_insn_line(struct repository *r, struct todo_item *item, - const char *buf, const char *bol, char *eol) +static int check_merge_commit_insn(enum todo_command command) +{ + switch(command) { + case TODO_PICK: + error(_("'%s' does not accept merge commits"), + todo_command_info[command].str); + advise_if_enabled(ADVICE_REBASE_TODO_ERROR, _( + /* + * TRANSLATORS: 'pick' and 'merge -C' should not be + * translated. + */ + "'pick' does not take a merge commit. If you wanted to\n" + "replay the merge, use 'merge -C' on the commit.")); + return -1; + + case TODO_REWORD: + error(_("'%s' does not accept merge commits"), + todo_command_info[command].str); + advise_if_enabled(ADVICE_REBASE_TODO_ERROR, _( + /* + * TRANSLATORS: 'reword' and 'merge -c' should not be + * translated. + */ + "'reword' does not take a merge commit. If you wanted to\n" + "replay the merge and reword the commit message, use\n" + "'merge -c' on the commit")); + return -1; + + case TODO_EDIT: + error(_("'%s' does not accept merge commits"), + todo_command_info[command].str); + advise_if_enabled(ADVICE_REBASE_TODO_ERROR, _( + /* + * TRANSLATORS: 'edit', 'merge -C' and 'break' should + * not be translated. + */ + "'edit' does not take a merge commit. If you wanted to\n" + "replay the merge, use 'merge -C' on the commit, and then\n" + "'break' to give the control back to you so that you can\n" + "do 'git commit --amend && git rebase --continue'.")); + return -1; + + case TODO_FIXUP: + case TODO_SQUASH: + return error(_("cannot squash merge commit into another commit")); + + case TODO_MERGE: + return 0; + + default: + BUG("unexpected todo_command"); + } +} + +static int parse_insn_line(struct repository *r, struct replay_opts *opts, + struct todo_item *item, const char *buf, + const char *bol, char *eol) { struct object_id commit_oid; char *end_of_object_name; @@ -2731,7 +2786,12 @@ static int parse_insn_line(struct repository *r, struct todo_item *item, return status; item->commit = lookup_commit_reference(r, &commit_oid); - return item->commit ? 0 : -1; + if (!item->commit) + return -1; + if (is_rebase_i(opts) && + item->commit->parents && item->commit->parents->next) + return check_merge_commit_insn(item->command); + return 0; } int sequencer_get_last_command(struct repository *r UNUSED, enum replay_action *action) @@ -2761,8 +2821,8 @@ int sequencer_get_last_command(struct repository *r UNUSED, enum replay_action * return ret; } -int todo_list_parse_insn_buffer(struct repository *r, char *buf, - struct todo_list *todo_list) +int todo_list_parse_insn_buffer(struct repository *r, struct replay_opts *opts, + char *buf, struct todo_list *todo_list) { struct todo_item *item; char *p = buf, *next_p; @@ -2780,7 +2840,7 @@ int todo_list_parse_insn_buffer(struct repository *r, char *buf, item = append_new_todo(todo_list); item->offset_in_buf = p - todo_list->buf.buf; - if (parse_insn_line(r, item, buf, p, eol)) { + if (parse_insn_line(r, opts, item, buf, p, eol)) { res = error(_("invalid line %d: %.*s"), i, (int)(eol - p), p); item->command = TODO_COMMENT + 1; @@ -2930,7 +2990,7 @@ static int read_populate_todo(struct repository *r, if (strbuf_read_file_or_whine(&todo_list->buf, todo_file) < 0) return -1; - res = todo_list_parse_insn_buffer(r, todo_list->buf.buf, todo_list); + res = todo_list_parse_insn_buffer(r, opts, todo_list->buf.buf, todo_list); if (res) { if (is_rebase_i(opts)) return error(_("please fix this using " @@ -2960,7 +3020,7 @@ static int read_populate_todo(struct repository *r, struct todo_list done = TODO_LIST_INIT; if (strbuf_read_file(&done.buf, rebase_path_done(), 0) > 0 && - !todo_list_parse_insn_buffer(r, done.buf.buf, &done)) + !todo_list_parse_insn_buffer(r, opts, done.buf.buf, &done)) todo_list->done_nr = count_commands(&done); else todo_list->done_nr = 0; @@ -5319,7 +5379,8 @@ int sequencer_continue(struct repository *r, struct replay_opts *opts) goto release_todo_list; if (file_exists(rebase_path_dropped())) { - if ((res = todo_list_check_against_backup(r, &todo_list))) + if ((res = todo_list_check_against_backup(r, opts, + &todo_list))) goto release_todo_list; unlink(rebase_path_dropped()); @@ -6363,7 +6424,7 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla return error(_("nothing to do")); } - res = edit_todo_list(r, todo_list, &new_todo, shortrevisions, + res = edit_todo_list(r, opts, todo_list, &new_todo, shortrevisions, shortonto, flags); if (res == -1) return -1; @@ -6391,7 +6452,7 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla strbuf_release(&buf2); /* Nothing is done yet, and we're reparsing, so let's reset the count */ new_todo.total_nr = 0; - if (todo_list_parse_insn_buffer(r, new_todo.buf.buf, &new_todo) < 0) + if (todo_list_parse_insn_buffer(r, opts, new_todo.buf.buf, &new_todo) < 0) BUG("invalid todo list after expanding IDs:\n%s", new_todo.buf.buf); diff --git a/sequencer.h b/sequencer.h index a309ddd712..304ba4b4d3 100644 --- a/sequencer.h +++ b/sequencer.h @@ -136,8 +136,8 @@ struct todo_list { .buf = STRBUF_INIT, \ } -int todo_list_parse_insn_buffer(struct repository *r, char *buf, - struct todo_list *todo_list); +int todo_list_parse_insn_buffer(struct repository *r, struct replay_opts *opts, + char *buf, struct todo_list *todo_list); int todo_list_write_to_file(struct repository *r, struct todo_list *todo_list, const char *file, const char *shortrevisions, const char *shortonto, int num, unsigned flags); diff --git a/server-info.c b/server-info.c index 6feaa457c5..37d1085982 100644 --- a/server-info.c +++ b/server-info.c @@ -13,6 +13,7 @@ #include "object-store-ll.h" #include "server-info.h" #include "strbuf.h" +#include "tempfile.h" struct update_info_ctx { FILE *cur_fp; @@ -75,9 +76,8 @@ static int update_info_file(char *path, int force) { char *tmp = mkpathdup("%s_XXXXXX", path); + struct tempfile *f = NULL; int ret = -1; - int fd = -1; - FILE *to_close; struct update_info_ctx uic = { .cur_fp = NULL, .old_fp = NULL, @@ -86,13 +86,12 @@ static int update_info_file(char *path, }; safe_create_leading_directories(path); - fd = git_mkstemp_mode(tmp, 0666); - if (fd < 0) + f = mks_tempfile_m(tmp, 0666); + if (!f) goto out; - to_close = uic.cur_fp = fdopen(fd, "w"); + uic.cur_fp = fdopen_tempfile(f, "w"); if (!uic.cur_fp) goto out; - fd = -1; /* no problem on ENOENT and old_fp == NULL, it's stale, now */ if (!force) @@ -121,27 +120,22 @@ static int update_info_file(char *path, } uic.cur_fp = NULL; - if (fclose(to_close)) - goto out; if (uic_is_stale(&uic)) { - if (adjust_shared_perm(tmp) < 0) + if (adjust_shared_perm(get_tempfile_path(f)) < 0) goto out; - if (rename(tmp, path) < 0) + if (rename_tempfile(&f, path) < 0) goto out; } else { - unlink(tmp); + delete_tempfile(&f); } ret = 0; out: if (ret) { error_errno("unable to update %s", path); - if (uic.cur_fp) - fclose(uic.cur_fp); - else if (fd >= 0) - close(fd); - unlink(tmp); + if (f) + delete_tempfile(&f); } free(tmp); if (uic.old_fp) @@ -49,7 +49,7 @@ static int abspath_part_inside_repo(char *path) size_t wtlen; char *path0; int off; - const char *work_tree = get_git_work_tree(); + const char *work_tree = precompose_string_if_needed(get_git_work_tree()); struct strbuf realpath = STRBUF_INIT; if (!work_tree) diff --git a/t/helper/test-bitmap.c b/t/helper/test-bitmap.c index af43ee1cb5..6af2b42678 100644 --- a/t/helper/test-bitmap.c +++ b/t/helper/test-bitmap.c @@ -13,21 +13,41 @@ static int bitmap_dump_hashes(void) return test_bitmap_hashes(the_repository); } +static int bitmap_dump_pseudo_merges(void) +{ + return test_bitmap_pseudo_merges(the_repository); +} + +static int bitmap_dump_pseudo_merge_commits(uint32_t n) +{ + return test_bitmap_pseudo_merge_commits(the_repository, n); +} + +static int bitmap_dump_pseudo_merge_objects(uint32_t n) +{ + return test_bitmap_pseudo_merge_objects(the_repository, n); +} + int cmd__bitmap(int argc, const char **argv) { setup_git_directory(); - if (argc != 2) - goto usage; - - if (!strcmp(argv[1], "list-commits")) + if (argc == 2 && !strcmp(argv[1], "list-commits")) return bitmap_list_commits(); - if (!strcmp(argv[1], "dump-hashes")) + if (argc == 2 && !strcmp(argv[1], "dump-hashes")) return bitmap_dump_hashes(); + if (argc == 2 && !strcmp(argv[1], "dump-pseudo-merges")) + return bitmap_dump_pseudo_merges(); + if (argc == 3 && !strcmp(argv[1], "dump-pseudo-merge-commits")) + return bitmap_dump_pseudo_merge_commits(atoi(argv[2])); + if (argc == 3 && !strcmp(argv[1], "dump-pseudo-merge-objects")) + return bitmap_dump_pseudo_merge_objects(atoi(argv[2])); -usage: usage("\ttest-tool bitmap list-commits\n" - "\ttest-tool bitmap dump-hashes"); + "\ttest-tool bitmap dump-hashes\n" + "\ttest-tool bitmap dump-pseudo-merges\n" + "\ttest-tool bitmap dump-pseudo-merge-commits <n>\n" + "\ttest-tool bitmap dump-pseudo-merge-objects <n>"); return -1; } diff --git a/t/helper/test-oidtree.c b/t/helper/test-oidtree.c deleted file mode 100644 index c7a1d4c642..0000000000 --- a/t/helper/test-oidtree.c +++ /dev/null @@ -1,54 +0,0 @@ -#include "test-tool.h" -#include "hex.h" -#include "oidtree.h" -#include "setup.h" -#include "strbuf.h" - -static enum cb_next print_oid(const struct object_id *oid, void *data UNUSED) -{ - puts(oid_to_hex(oid)); - return CB_CONTINUE; -} - -int cmd__oidtree(int argc UNUSED, const char **argv UNUSED) -{ - struct oidtree ot; - struct strbuf line = STRBUF_INIT; - int nongit_ok; - int algo = GIT_HASH_UNKNOWN; - - oidtree_init(&ot); - setup_git_directory_gently(&nongit_ok); - - while (strbuf_getline(&line, stdin) != EOF) { - const char *arg; - struct object_id oid; - - if (skip_prefix(line.buf, "insert ", &arg)) { - if (get_oid_hex_any(arg, &oid) == GIT_HASH_UNKNOWN) - die("insert not a hexadecimal oid: %s", arg); - algo = oid.algo; - oidtree_insert(&ot, &oid); - } else if (skip_prefix(line.buf, "contains ", &arg)) { - if (get_oid_hex(arg, &oid)) - die("contains not a hexadecimal oid: %s", arg); - printf("%d\n", oidtree_contains(&ot, &oid)); - } else if (skip_prefix(line.buf, "each ", &arg)) { - char buf[GIT_MAX_HEXSZ + 1] = { '0' }; - memset(&oid, 0, sizeof(oid)); - memcpy(buf, arg, strlen(arg)); - buf[hash_algos[algo].hexsz] = '\0'; - get_oid_hex_any(buf, &oid); - oid.algo = algo; - oidtree_each(&ot, &oid, strlen(arg), print_oid, NULL); - } else if (!strcmp(line.buf, "clear")) { - oidtree_clear(&ot); - } else { - die("unknown command: %s", line.buf); - } - } - - strbuf_release(&line); - - return 0; -} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 1304336fe8..93436a82ae 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -45,7 +45,6 @@ static struct test_cmd cmds[] = { { "mktemp", cmd__mktemp }, { "oid-array", cmd__oid_array }, { "oidmap", cmd__oidmap }, - { "oidtree", cmd__oidtree }, { "online-cpus", cmd__online_cpus }, { "pack-mtimes", cmd__pack_mtimes }, { "parse-options", cmd__parse_options }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index 93a125d21a..d9033d14e1 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -38,7 +38,6 @@ int cmd__match_trees(int argc, const char **argv); int cmd__mergesort(int argc, const char **argv); int cmd__mktemp(int argc, const char **argv); int cmd__oidmap(int argc, const char **argv); -int cmd__oidtree(int argc, const char **argv); int cmd__online_cpus(int argc, const char **argv); int cmd__pack_mtimes(int argc, const char **argv); int cmd__parse_options(int argc, const char **argv); diff --git a/t/perf/p5333-pseudo-merge-bitmaps.sh b/t/perf/p5333-pseudo-merge-bitmaps.sh new file mode 100755 index 0000000000..2e8b1d2635 --- /dev/null +++ b/t/perf/p5333-pseudo-merge-bitmaps.sh @@ -0,0 +1,32 @@ +#!/bin/sh + +test_description='pseudo-merge bitmaps' +. ./perf-lib.sh + +test_perf_large_repo + +test_expect_success 'setup' ' + git \ + -c bitmapPseudoMerge.all.pattern="refs/" \ + -c bitmapPseudoMerge.all.threshold=now \ + -c bitmapPseudoMerge.all.stableThreshold=never \ + -c bitmapPseudoMerge.all.maxMerges=64 \ + -c pack.writeBitmapLookupTable=true \ + repack -adb +' + +test_perf 'git rev-list --count --all --objects (no bitmaps)' ' + git rev-list --objects --all +' + +test_perf 'git rev-list --count --all --objects (no pseudo-merges)' ' + GIT_TEST_USE_PSEUDO_MERGES=0 \ + git rev-list --objects --all --use-bitmap-index +' + +test_perf 'git rev-list --count --all --objects (with pseudo-merges)' ' + GIT_TEST_USE_PSEUDO_MERGES=1 \ + git rev-list --objects --all --use-bitmap-index +' + +test_done diff --git a/t/t0069-oidtree.sh b/t/t0069-oidtree.sh deleted file mode 100755 index 889db50818..0000000000 --- a/t/t0069-oidtree.sh +++ /dev/null @@ -1,50 +0,0 @@ -#!/bin/sh - -test_description='basic tests for the oidtree implementation' -TEST_PASSES_SANITIZE_LEAK=true -. ./test-lib.sh - -maxhexsz=$(test_oid hexsz) -echoid () { - prefix="${1:+$1 }" - shift - while test $# -gt 0 - do - shortoid="$1" - shift - difference=$(($maxhexsz - ${#shortoid})) - printf "%s%s%0${difference}d\\n" "$prefix" "$shortoid" "0" - done -} - -test_expect_success 'oidtree insert and contains' ' - cat >expect <<-\EOF && - 0 - 0 - 0 - 1 - 1 - 0 - EOF - { - echoid insert 444 1 2 3 4 5 a b c d e && - echoid contains 44 441 440 444 4440 4444 && - echo clear - } | test-tool oidtree >actual && - test_cmp expect actual -' - -test_expect_success 'oidtree each' ' - echoid "" 123 321 321 >expect && - { - echoid insert f 9 8 123 321 a b c d e && - echo each 12300 && - echo each 3211 && - echo each 3210 && - echo each 32100 && - echo clear - } | test-tool oidtree >actual && - test_cmp expect actual -' - -test_done diff --git a/t/t0600-reffiles-backend.sh b/t/t0600-reffiles-backend.sh index 92f570313d..b2a771ff2b 100755 --- a/t/t0600-reffiles-backend.sh +++ b/t/t0600-reffiles-backend.sh @@ -468,4 +468,36 @@ test_expect_success POSIXPERM 'git reflog expire honors core.sharedRepository' ' esac ' +test_expect_success SYMLINKS 'symref transaction supports symlinks' ' + test_when_finished "git symbolic-ref -d TEST_SYMREF_HEAD" && + git update-ref refs/heads/new @ && + test_config core.prefersymlinkrefs true && + cat >stdin <<-EOF && + start + symref-create TEST_SYMREF_HEAD refs/heads/new + prepare + commit + EOF + git update-ref --no-deref --stdin <stdin && + test_path_is_symlink .git/TEST_SYMREF_HEAD && + test "$(test_readlink .git/TEST_SYMREF_HEAD)" = refs/heads/new +' + +test_expect_success 'symref transaction supports false symlink config' ' + test_when_finished "git symbolic-ref -d TEST_SYMREF_HEAD" && + git update-ref refs/heads/new @ && + test_config core.prefersymlinkrefs false && + cat >stdin <<-EOF && + start + symref-create TEST_SYMREF_HEAD refs/heads/new + prepare + commit + EOF + git update-ref --no-deref --stdin <stdin && + test_path_is_file .git/TEST_SYMREF_HEAD && + git symbolic-ref TEST_SYMREF_HEAD >actual && + echo refs/heads/new >expect && + test_cmp expect actual +' + test_done diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh index bbee2783ab..eb1691860d 100755 --- a/t/t1400-update-ref.sh +++ b/t/t1400-update-ref.sh @@ -892,17 +892,23 @@ test_expect_success 'stdin update/create/verify combination works' ' ' test_expect_success 'stdin verify succeeds for correct value' ' + test-tool ref-store main for-each-reflog-ent $m >before && git rev-parse $m >expect && echo "verify $m $m" >stdin && git update-ref --stdin <stdin && git rev-parse $m >actual && - test_cmp expect actual + test_cmp expect actual && + test-tool ref-store main for-each-reflog-ent $m >after && + test_cmp before after ' test_expect_success 'stdin verify succeeds for missing reference' ' + test-tool ref-store main for-each-reflog-ent $m >before && echo "verify refs/heads/missing $Z" >stdin && git update-ref --stdin <stdin && - test_must_fail git rev-parse --verify -q refs/heads/missing + test_must_fail git rev-parse --verify -q refs/heads/missing && + test-tool ref-store main for-each-reflog-ent $m >after && + test_cmp before after ' test_expect_success 'stdin verify treats no value as missing' ' @@ -1356,6 +1362,7 @@ test_expect_success 'fails with duplicate HEAD update' ' ' test_expect_success 'fails with duplicate ref update via symref' ' + test_when_finished "git symbolic-ref -d refs/heads/symref2" && git branch target2 $A && git symbolic-ref refs/heads/symref2 refs/heads/target2 && cat >stdin <<-EOF && @@ -1643,4 +1650,423 @@ test_expect_success PIPE 'transaction flushes status updates' ' test_cmp expected actual ' +format_command () { + if test "$1" = "-z" + then + shift + printf "$F" "$@" + else + echo "$@" + fi +} + +for type in "" "-z" +do + + test_expect_success "stdin $type symref-verify fails without --no-deref" ' + git symbolic-ref refs/heads/symref $a && + format_command $type "symref-verify refs/heads/symref" "$a" >stdin && + test_must_fail git update-ref --stdin $type <stdin 2>err && + grep "fatal: symref-verify: cannot operate with deref mode" err + ' + + test_expect_success "stdin $type symref-verify fails with too many arguments" ' + format_command $type "symref-verify refs/heads/symref" "$a" "$a" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin 2>err && + if test "$type" = "-z" + then + grep "fatal: unknown command: $a" err + else + grep "fatal: symref-verify refs/heads/symref: extra input: $a" err + fi + ' + + test_expect_success "stdin $type symref-verify succeeds for correct value" ' + git symbolic-ref refs/heads/symref >expect && + test-tool ref-store main for-each-reflog-ent refs/heads/symref >before && + format_command $type "symref-verify refs/heads/symref" "$a" >stdin && + git update-ref --stdin $type --no-deref <stdin && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual && + test-tool ref-store main for-each-reflog-ent refs/heads/symref >after && + test_cmp before after + ' + + test_expect_success "stdin $type symref-verify fails with no value" ' + git symbolic-ref refs/heads/symref >expect && + format_command $type "symref-verify refs/heads/symref" "" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin + ' + + test_expect_success "stdin $type symref-verify succeeds for dangling reference" ' + test_when_finished "git symbolic-ref -d refs/heads/symref2" && + test_must_fail git symbolic-ref refs/heads/nonexistent && + git symbolic-ref refs/heads/symref2 refs/heads/nonexistent && + format_command $type "symref-verify refs/heads/symref2" "refs/heads/nonexistent" >stdin && + git update-ref --stdin $type --no-deref <stdin + ' + + test_expect_success "stdin $type symref-verify fails for missing reference" ' + test-tool ref-store main for-each-reflog-ent refs/heads/symref >before && + format_command $type "symref-verify refs/heads/missing" "refs/heads/unknown" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin 2>err && + grep "fatal: cannot lock ref ${SQ}refs/heads/missing${SQ}: unable to resolve reference ${SQ}refs/heads/missing${SQ}" err && + test_must_fail git rev-parse --verify -q refs/heads/missing && + test-tool ref-store main for-each-reflog-ent refs/heads/symref >after && + test_cmp before after + ' + + test_expect_success "stdin $type symref-verify fails for wrong value" ' + git symbolic-ref refs/heads/symref >expect && + format_command $type "symref-verify refs/heads/symref" "$b" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-verify fails for mistaken null value" ' + git symbolic-ref refs/heads/symref >expect && + format_command $type "symref-verify refs/heads/symref" "$Z" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-delete fails without --no-deref" ' + git symbolic-ref refs/heads/symref $a && + format_command $type "symref-delete refs/heads/symref" "$a" >stdin && + test_must_fail git update-ref --stdin $type <stdin 2>err && + grep "fatal: symref-delete: cannot operate with deref mode" err + ' + + test_expect_success "stdin $type symref-delete fails with no ref" ' + format_command $type "symref-delete " >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin 2>err && + grep "fatal: symref-delete: missing <ref>" err + ' + + test_expect_success "stdin $type symref-delete fails deleting regular ref" ' + test_when_finished "git update-ref -d refs/heads/regularref" && + git update-ref refs/heads/regularref $a && + format_command $type "symref-delete refs/heads/regularref" "$a" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin 2>err && + grep "fatal: cannot lock ref ${SQ}refs/heads/regularref${SQ}: expected symref with target ${SQ}$a${SQ}: but is a regular ref" err + ' + + test_expect_success "stdin $type symref-delete fails with too many arguments" ' + format_command $type "symref-delete refs/heads/symref" "$a" "$a" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin 2>err && + if test "$type" = "-z" + then + grep "fatal: unknown command: $a" err + else + grep "fatal: symref-delete refs/heads/symref: extra input: $a" err + fi + ' + + test_expect_success "stdin $type symref-delete fails with wrong old value" ' + format_command $type "symref-delete refs/heads/symref" "$m" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin 2>err && + grep "fatal: verifying symref target: ${SQ}refs/heads/symref${SQ}: is at $a but expected refs/heads/main" err && + git symbolic-ref refs/heads/symref >expect && + echo $a >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-delete works with right old value" ' + format_command $type "symref-delete refs/heads/symref" "$a" >stdin && + git update-ref --stdin $type --no-deref <stdin && + test_must_fail git rev-parse --verify -q refs/heads/symref + ' + + test_expect_success "stdin $type symref-delete works with empty old value" ' + git symbolic-ref refs/heads/symref $a >stdin && + format_command $type "symref-delete refs/heads/symref" "" >stdin && + git update-ref --stdin $type --no-deref <stdin && + test_must_fail git rev-parse --verify -q $b + ' + + test_expect_success "stdin $type symref-delete succeeds for dangling reference" ' + test_must_fail git symbolic-ref refs/heads/nonexistent && + git symbolic-ref refs/heads/symref2 refs/heads/nonexistent && + format_command $type "symref-delete refs/heads/symref2" "refs/heads/nonexistent" >stdin && + git update-ref --stdin $type --no-deref <stdin && + test_must_fail git symbolic-ref -d refs/heads/symref2 + ' + + test_expect_success "stdin $type symref-delete deletes regular ref without target" ' + git update-ref refs/heads/regularref $a && + format_command $type "symref-delete refs/heads/regularref" >stdin && + git update-ref --stdin $type --no-deref <stdin + ' + + test_expect_success "stdin $type symref-create fails with too many arguments" ' + format_command $type "symref-create refs/heads/symref" "$a" "$a" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin 2>err && + if test "$type" = "-z" + then + grep "fatal: unknown command: $a" err + else + grep "fatal: symref-create refs/heads/symref: extra input: $a" err + fi + ' + + test_expect_success "stdin $type symref-create fails with no target" ' + format_command $type "symref-create refs/heads/symref" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin + ' + + test_expect_success "stdin $type symref-create fails with empty target" ' + format_command $type "symref-create refs/heads/symref" "" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin + ' + + test_expect_success "stdin $type symref-create works" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + format_command $type "symref-create refs/heads/symref" "$a" >stdin && + git update-ref --stdin $type --no-deref <stdin && + git symbolic-ref refs/heads/symref >expect && + echo $a >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-create works with --no-deref" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + format_command $type "symref-create refs/heads/symref" "$a" && + git update-ref --stdin $type <stdin 2>err + ' + + test_expect_success "stdin $type create dangling symref ref works" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + format_command $type "symref-create refs/heads/symref" "refs/heads/unkown" >stdin && + git update-ref --stdin $type --no-deref <stdin && + git symbolic-ref refs/heads/symref >expect && + echo refs/heads/unkown >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-create does not create reflogs by default" ' + test_when_finished "git symbolic-ref -d refs/symref" && + format_command $type "symref-create refs/symref" "$a" >stdin && + git update-ref --stdin $type --no-deref <stdin && + git symbolic-ref refs/symref >expect && + echo $a >actual && + test_cmp expect actual && + test_must_fail git reflog exists refs/symref + ' + + test_expect_success "stdin $type symref-create reflogs with --create-reflog" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + format_command $type "symref-create refs/heads/symref" "$a" >stdin && + git update-ref --create-reflog --stdin $type --no-deref <stdin && + git symbolic-ref refs/heads/symref >expect && + echo $a >actual && + test_cmp expect actual && + git reflog exists refs/heads/symref + ' + + test_expect_success "stdin $type symref-update fails with too many arguments" ' + format_command $type "symref-update refs/heads/symref" "$a" "ref" "$a" "$a" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin 2>err && + if test "$type" = "-z" + then + grep "fatal: unknown command: $a" err + else + grep "fatal: symref-update refs/heads/symref: extra input: $a" err + fi + ' + + test_expect_success "stdin $type symref-update fails with wrong old value argument" ' + format_command $type "symref-update refs/heads/symref" "$a" "foo" "$a" "$a" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin 2>err && + grep "fatal: symref-update refs/heads/symref: invalid arg ${SQ}foo${SQ} for old value" err + ' + + test_expect_success "stdin $type symref-update creates with zero old value" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + format_command $type "symref-update refs/heads/symref" "$a" "oid" "$Z" >stdin && + git update-ref --stdin $type --no-deref <stdin && + echo $a >expect && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update creates with no old value" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + format_command $type "symref-update refs/heads/symref" "$a" >stdin && + git update-ref --stdin $type --no-deref <stdin && + echo $a >expect && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update creates dangling" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + test_must_fail git rev-parse refs/heads/nonexistent && + format_command $type "symref-update refs/heads/symref" "refs/heads/nonexistent" >stdin && + git update-ref --stdin $type --no-deref <stdin && + echo refs/heads/nonexistent >expect && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update fails with wrong old value" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + git symbolic-ref refs/heads/symref $a && + format_command $type "symref-update refs/heads/symref" "$m" "ref" "$b" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin 2>err && + grep "fatal: verifying symref target: ${SQ}refs/heads/symref${SQ}: is at $a but expected $b" err && + test_must_fail git rev-parse --verify -q $c + ' + + test_expect_success "stdin $type symref-update updates dangling ref" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + test_must_fail git rev-parse refs/heads/nonexistent && + git symbolic-ref refs/heads/symref refs/heads/nonexistent && + format_command $type "symref-update refs/heads/symref" "$a" >stdin && + git update-ref --stdin $type --no-deref <stdin && + echo $a >expect && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update updates dangling ref with old value" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + test_must_fail git rev-parse refs/heads/nonexistent && + git symbolic-ref refs/heads/symref refs/heads/nonexistent && + format_command $type "symref-update refs/heads/symref" "$a" "ref" "refs/heads/nonexistent" >stdin && + git update-ref --stdin $type --no-deref <stdin && + echo $a >expect && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update fails update dangling ref with wrong old value" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + test_must_fail git rev-parse refs/heads/nonexistent && + git symbolic-ref refs/heads/symref refs/heads/nonexistent && + format_command $type "symref-update refs/heads/symref" "$a" "ref" "refs/heads/wrongref" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin && + echo refs/heads/nonexistent >expect && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update works with right old value" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + git symbolic-ref refs/heads/symref $a && + format_command $type "symref-update refs/heads/symref" "$m" "ref" "$a" >stdin && + git update-ref --stdin $type --no-deref <stdin && + echo $m >expect && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update works with no old value" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + git symbolic-ref refs/heads/symref $a && + format_command $type "symref-update refs/heads/symref" "$m" >stdin && + git update-ref --stdin $type --no-deref <stdin && + echo $m >expect && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update fails with empty old ref-target" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + git symbolic-ref refs/heads/symref $a && + format_command $type "symref-update refs/heads/symref" "$m" "ref" "" >stdin && + test_must_fail git update-ref --stdin $type --no-deref <stdin && + echo $a >expect && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update creates (with deref)" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + format_command $type "symref-update refs/heads/symref" "$a" >stdin && + git update-ref --stdin $type <stdin && + echo $a >expect && + git symbolic-ref --no-recurse refs/heads/symref >actual && + test_cmp expect actual && + test-tool ref-store main for-each-reflog-ent refs/heads/symref >actual && + grep "$Z $(git rev-parse $a)" actual + ' + + test_expect_success "stdin $type symref-update regular ref to symref with correct old-oid" ' + test_when_finished "git symbolic-ref -d --no-recurse refs/heads/regularref" && + git update-ref --no-deref refs/heads/regularref $a && + format_command $type "symref-update refs/heads/regularref" "$a" "oid" "$(git rev-parse $a)" >stdin && + git update-ref --stdin $type <stdin && + echo $a >expect && + git symbolic-ref --no-recurse refs/heads/regularref >actual && + test_cmp expect actual && + test-tool ref-store main for-each-reflog-ent refs/heads/regularref >actual && + grep "$(git rev-parse $a) $(git rev-parse $a)" actual + ' + + test_expect_success "stdin $type symref-update regular ref to symref fails with wrong old-oid" ' + test_when_finished "git update-ref -d refs/heads/regularref" && + git update-ref --no-deref refs/heads/regularref $a && + format_command $type "symref-update refs/heads/regularref" "$a" "oid" "$(git rev-parse refs/heads/target2)" >stdin && + test_must_fail git update-ref --stdin $type <stdin 2>err && + grep "fatal: cannot lock ref ${SQ}refs/heads/regularref${SQ}: is at $(git rev-parse $a) but expected $(git rev-parse refs/heads/target2)" err && + echo $(git rev-parse $a) >expect && + git rev-parse refs/heads/regularref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update regular ref to symref fails with invalid old-oid" ' + test_when_finished "git update-ref -d refs/heads/regularref" && + git update-ref --no-deref refs/heads/regularref $a && + format_command $type "symref-update refs/heads/regularref" "$a" "oid" "not-a-ref-oid" >stdin && + test_must_fail git update-ref --stdin $type <stdin 2>err && + grep "fatal: symref-update refs/heads/regularref: invalid oid: not-a-ref-oid" err && + echo $(git rev-parse $a) >expect && + git rev-parse refs/heads/regularref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update existing symref with zero old-oid" ' + test_when_finished "git symbolic-ref -d --no-recurse refs/heads/symref" && + git symbolic-ref refs/heads/symref refs/heads/target2 && + format_command $type "symref-update refs/heads/symref" "$a" "oid" "$Z" >stdin && + test_must_fail git update-ref --stdin $type <stdin 2>err && + grep "fatal: cannot lock ref ${SQ}refs/heads/symref${SQ}: reference already exists" err && + echo refs/heads/target2 >expect && + git symbolic-ref refs/heads/symref >actual && + test_cmp expect actual + ' + + test_expect_success "stdin $type symref-update regular ref to symref (with deref)" ' + test_when_finished "git symbolic-ref -d refs/heads/symref" && + test_when_finished "git update-ref -d --no-deref refs/heads/symref2" && + git update-ref refs/heads/symref2 $a && + git symbolic-ref --no-recurse refs/heads/symref refs/heads/symref2 && + format_command $type "symref-update refs/heads/symref" "$a" >stdin && + git update-ref $type --stdin <stdin && + echo $a >expect && + git symbolic-ref --no-recurse refs/heads/symref2 >actual && + test_cmp expect actual && + echo refs/heads/symref2 >expect && + git symbolic-ref --no-recurse refs/heads/symref >actual && + test_cmp expect actual && + test-tool ref-store main for-each-reflog-ent refs/heads/symref >actual && + grep "$(git rev-parse $a) $(git rev-parse $a)" actual + ' + + test_expect_success "stdin $type symref-update regular ref to symref" ' + test_when_finished "git symbolic-ref -d --no-recurse refs/heads/regularref" && + git update-ref --no-deref refs/heads/regularref $a && + format_command $type "symref-update refs/heads/regularref" "$a" >stdin && + git update-ref $type --stdin <stdin && + echo $a >expect && + git symbolic-ref --no-recurse refs/heads/regularref >actual && + test_cmp expect actual && + test-tool ref-store main for-each-reflog-ent refs/heads/regularref >actual && + grep "$(git rev-parse $a) $(git rev-parse $a)" actual + ' + +done + test_done diff --git a/t/t1403-show-ref.sh b/t/t1403-show-ref.sh index 33fb7a38ff..403f6b8f7d 100755 --- a/t/t1403-show-ref.sh +++ b/t/t1403-show-ref.sh @@ -121,13 +121,13 @@ test_expect_success 'show-ref -d' ' ' -test_expect_success 'show-ref --heads, --tags, --head, pattern' ' +test_expect_success 'show-ref --branches, --tags, --head, pattern' ' for branch in B main side do echo $(git rev-parse refs/heads/$branch) refs/heads/$branch || return 1 - done >expect.heads && - git show-ref --heads >actual && - test_cmp expect.heads actual && + done >expect.branches && + git show-ref --branches >actual && + test_cmp expect.branches actual && for tag in A B C do @@ -136,15 +136,15 @@ test_expect_success 'show-ref --heads, --tags, --head, pattern' ' git show-ref --tags >actual && test_cmp expect.tags actual && - cat expect.heads expect.tags >expect && - git show-ref --heads --tags >actual && + cat expect.branches expect.tags >expect && + git show-ref --branches --tags >actual && test_cmp expect actual && { echo $(git rev-parse HEAD) HEAD && - cat expect.heads expect.tags + cat expect.branches expect.tags } >expect && - git show-ref --heads --tags --head >actual && + git show-ref --branches --tags --head >actual && test_cmp expect actual && { @@ -165,6 +165,14 @@ test_expect_success 'show-ref --heads, --tags, --head, pattern' ' test_cmp expect actual ' +test_expect_success 'show-ref --heads is deprecated and hidden' ' + test_expect_code 129 git show-ref -h >short-help && + test_grep ! -e --heads short-help && + git show-ref --heads >actual 2>warning && + test_grep ! deprecated warning && + test_cmp expect.branches actual +' + test_expect_success 'show-ref --verify HEAD' ' echo $(git rev-parse HEAD) HEAD >expect && git show-ref --verify HEAD >actual && diff --git a/t/t1416-ref-transaction-hooks.sh b/t/t1416-ref-transaction-hooks.sh index 067fd57290..5a812ca3c0 100755 --- a/t/t1416-ref-transaction-hooks.sh +++ b/t/t1416-ref-transaction-hooks.sh @@ -157,4 +157,58 @@ test_expect_success 'hook captures git-symbolic-ref updates' ' test_cmp expect actual ' +test_expect_success 'hook gets all queued symref updates' ' + test_when_finished "rm actual" && + + git update-ref refs/heads/branch $POST_OID && + git symbolic-ref refs/heads/symref refs/heads/main && + git symbolic-ref refs/heads/symrefd refs/heads/main && + git symbolic-ref refs/heads/symrefu refs/heads/main && + + test_hook reference-transaction <<-\EOF && + echo "$*" >>actual + while read -r line + do + printf "%s\n" "$line" + done >>actual + EOF + + # In the files backend, "delete" also triggers an additional transaction + # update on the packed-refs backend, which constitutes additional reflog + # entries. + if test_have_prereq REFFILES + then + cat >expect <<-EOF + aborted + $ZERO_OID $ZERO_OID refs/heads/symrefd + EOF + else + >expect + fi && + + cat >>expect <<-EOF && + prepared + ref:refs/heads/main $ZERO_OID refs/heads/symref + ref:refs/heads/main $ZERO_OID refs/heads/symrefd + $ZERO_OID ref:refs/heads/main refs/heads/symrefc + ref:refs/heads/main ref:refs/heads/branch refs/heads/symrefu + committed + ref:refs/heads/main $ZERO_OID refs/heads/symref + ref:refs/heads/main $ZERO_OID refs/heads/symrefd + $ZERO_OID ref:refs/heads/main refs/heads/symrefc + ref:refs/heads/main ref:refs/heads/branch refs/heads/symrefu + EOF + + git update-ref --no-deref --stdin <<-EOF && + start + symref-verify refs/heads/symref refs/heads/main + symref-delete refs/heads/symrefd refs/heads/main + symref-create refs/heads/symrefc refs/heads/main + symref-update refs/heads/symrefu refs/heads/branch ref refs/heads/main + prepare + commit + EOF + test_cmp expect actual +' + test_done diff --git a/t/t3206-range-diff.sh b/t/t3206-range-diff.sh index 7b05bf3961..a767c3520e 100755 --- a/t/t3206-range-diff.sh +++ b/t/t3206-range-diff.sh @@ -545,6 +545,20 @@ do ' done +test_expect_success "--range-diff implies --cover-letter for multi-patch series" ' + test_when_finished "rm -f v2-000?-*" && + git format-patch -v2 --range-diff=topic main..unmodified && + test_grep "^Range-diff against v1:$" v2-0000-cover-letter.patch +' + +test_expect_success "explicit --no-cover-letter defeats implied --cover-letter" ' + test_when_finished "rm -f v2-000?-*" && + test_must_fail git format-patch --no-cover-letter \ + -v2 --range-diff=topic main..unmodified && + test_must_fail git -c format.coverLetter=no format-patch \ + -v2 --range-diff=topic main..unmodified +' + test_expect_success 'format-patch --range-diff as commentary' ' git format-patch --range-diff=HEAD~1 HEAD~1 >actual && test_when_finished "rm 0001-*" && diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh index d1bead61fa..f92baad138 100755 --- a/t/t3404-rebase-interactive.sh +++ b/t/t3404-rebase-interactive.sh @@ -2215,6 +2215,51 @@ test_expect_success 'bad labels and refs rejected when parsing todo list' ' test_path_is_missing execed ' +test_expect_success 'non-merge commands reject merge commits' ' + test_when_finished "test_might_fail git rebase --abort" && + git checkout E && + git merge I && + oid=$(git rev-parse HEAD) && + cat >todo <<-EOF && + pick $oid + reword $oid + edit $oid + fixup $oid + squash $oid + EOF + ( + set_replace_editor todo && + test_must_fail git rebase -i HEAD 2>actual + ) && + cat >expect <<-EOF && + error: ${SQ}pick${SQ} does not accept merge commits + hint: ${SQ}pick${SQ} does not take a merge commit. If you wanted to + hint: replay the merge, use ${SQ}merge -C${SQ} on the commit. + hint: Disable this message with "git config advice.rebaseTodoError false" + error: invalid line 1: pick $oid + error: ${SQ}reword${SQ} does not accept merge commits + hint: ${SQ}reword${SQ} does not take a merge commit. If you wanted to + hint: replay the merge and reword the commit message, use + hint: ${SQ}merge -c${SQ} on the commit + hint: Disable this message with "git config advice.rebaseTodoError false" + error: invalid line 2: reword $oid + error: ${SQ}edit${SQ} does not accept merge commits + hint: ${SQ}edit${SQ} does not take a merge commit. If you wanted to + hint: replay the merge, use ${SQ}merge -C${SQ} on the commit, and then + hint: ${SQ}break${SQ} to give the control back to you so that you can + hint: do ${SQ}git commit --amend && git rebase --continue${SQ}. + hint: Disable this message with "git config advice.rebaseTodoError false" + error: invalid line 3: edit $oid + error: cannot squash merge commit into another commit + error: invalid line 4: fixup $oid + error: cannot squash merge commit into another commit + error: invalid line 5: squash $oid + You can fix this with ${SQ}git rebase --edit-todo${SQ} and then run ${SQ}git rebase --continue${SQ}. + Or you can abort the rebase with ${SQ}git rebase --abort${SQ}. + EOF + test_cmp expect actual +' + # This must be the last test in this file test_expect_success '$EDITOR and friends are unchanged' ' test_editor_unchanged diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh index 6624a4f7c0..5d78868ac1 100755 --- a/t/t3701-add-interactive.sh +++ b/t/t3701-add-interactive.sh @@ -43,21 +43,6 @@ force_color () { ) } -test_expect_success 'warn about add.interactive.useBuiltin' ' - cat >expect <<-\EOF && - warning: the add.interactive.useBuiltin setting has been removed! - See its entry in '\''git help config'\'' for details. - EOF - echo "No changes." >expect.out && - - for v in = =true =false - do - git -c "add.interactive.useBuiltin$v" add -p >out 2>actual && - test_cmp expect.out out && - test_cmp expect actual || return 1 - done -' - test_expect_success 'unknown command' ' test_when_finished "git reset --hard; rm -f command" && echo W >command && diff --git a/t/t3910-mac-os-precompose.sh b/t/t3910-mac-os-precompose.sh index 898267a6bd..6d5918c8fe 100755 --- a/t/t3910-mac-os-precompose.sh +++ b/t/t3910-mac-os-precompose.sh @@ -37,6 +37,27 @@ Alongc=$Alongc$Alongc$Alongc$Alongc$Alongc #50 Byte Alongc=$Alongc$Alongc$Alongc$Alongc$Alongc #250 Byte Alongc=$Alongc$AEligatu$AEligatu #254 Byte + +ls_files_nfc_nfd () { + test_when_finished "git config --global --unset core.precomposeunicode" && + prglbl=$1 + prlocl=$2 + aumlcreat=$3 + aumllist=$4 + git config --global core.precomposeunicode $prglbl && + ( + rm -rf .git && + mkdir -p "somewhere/$prglbl/$prlocl/$aumlcreat" && + mypwd=$PWD && + cd "somewhere/$prglbl/$prlocl/$aumlcreat" && + git init && + git config core.precomposeunicode $prlocl && + git --literal-pathspecs ls-files "$mypwd/somewhere/$prglbl/$prlocl/$aumllist" 2>err && + >expected && + test_cmp expected err + ) +} + test_expect_success "detect if nfd needed" ' precomposeunicode=$(git config core.precomposeunicode) && test "$precomposeunicode" = true && @@ -211,8 +232,8 @@ test_expect_success "unicode decomposed: git restore -p . " ' ' # Test if the global core.precomposeunicode stops autosensing -# Must be the last test case test_expect_success "respect git config --global core.precomposeunicode" ' + test_when_finished "git config --global --unset core.precomposeunicode" && git config --global core.precomposeunicode true && rm -rf .git && git init && @@ -220,4 +241,20 @@ test_expect_success "respect git config --global core.precomposeunicode" ' test "$precomposeunicode" = "true" ' +test_expect_success "ls-files false false nfd nfd" ' + ls_files_nfc_nfd false false $Adiarnfd $Adiarnfd +' + +test_expect_success "ls-files false true nfd nfd" ' + ls_files_nfc_nfd false true $Adiarnfd $Adiarnfd +' + +test_expect_success "ls-files true false nfd nfd" ' + ls_files_nfc_nfd true false $Adiarnfd $Adiarnfd +' + +test_expect_success "ls-files true true nfd nfd" ' + ls_files_nfc_nfd true true $Adiarnfd $Adiarnfd +' + test_done diff --git a/t/t4014-format-patch.sh b/t/t4014-format-patch.sh index de039825a9..884f83fb8a 100755 --- a/t/t4014-format-patch.sh +++ b/t/t4014-format-patch.sh @@ -820,8 +820,8 @@ test_expect_success 'format-patch --notes --signoff' ' ' test_expect_success 'format-patch notes output control' ' + test_when_finished "git notes remove HEAD || :" && git notes add -m "notes config message" HEAD && - test_when_finished git notes remove HEAD && git format-patch -1 --stdout >out && ! grep "notes config message" out && @@ -848,10 +848,10 @@ test_expect_success 'format-patch notes output control' ' ' test_expect_success 'format-patch with multiple notes refs' ' + test_when_finished "git notes --ref note1 remove HEAD; + git notes --ref note2 remove HEAD || :" && git notes --ref note1 add -m "this is note 1" HEAD && - test_when_finished git notes --ref note1 remove HEAD && git notes --ref note2 add -m "this is note 2" HEAD && - test_when_finished git notes --ref note2 remove HEAD && git format-patch -1 --stdout >out && ! grep "this is note 1" out && @@ -892,10 +892,10 @@ test_expect_success 'format-patch with multiple notes refs' ' test_expect_success 'format-patch with multiple notes refs in config' ' test_when_finished "test_unconfig format.notes" && + test_when_finished "git notes --ref note1 remove HEAD; + git notes --ref note2 remove HEAD || :" && git notes --ref note1 add -m "this is note 1" HEAD && - test_when_finished git notes --ref note1 remove HEAD && git notes --ref note2 add -m "this is note 2" HEAD && - test_when_finished git notes --ref note2 remove HEAD && git config format.notes note1 && git format-patch -1 --stdout >out && @@ -2516,6 +2516,21 @@ test_expect_success 'range-diff: solo-patch' ' test_cmp expect actual ' +test_expect_success 'interdiff: multi-patch, implicit --cover-letter' ' + test_when_finished "rm -f v23-0*.patch" && + git format-patch --interdiff=boop~2 -2 -v23 && + test_grep "^Interdiff against v22:$" v23-0000-cover-letter.patch && + test_cmp expect actual +' + +test_expect_success 'interdiff: explicit --no-cover-letter defeats implied --cover-letter' ' + test_when_finished "rm -f v23-0*.patch" && + test_must_fail git format-patch --no-cover-letter \ + --interdiff=boop~2 -2 -v23 && + test_must_fail git -c format.coverLetter=no format-patch \ + --interdiff=boop~2 -2 -v23 +' + test_expect_success 'format-patch does not respect diff.noprefix' ' git -c diff.noprefix format-patch -1 --stdout >actual && grep "^--- a/blorp" actual diff --git a/t/t4020-diff-external.sh b/t/t4020-diff-external.sh index fdd865f7c3..3baa52a9bf 100755 --- a/t/t4020-diff-external.sh +++ b/t/t4020-diff-external.sh @@ -172,6 +172,72 @@ test_expect_success 'no diff with -diff' ' grep Binary out ' +check_external_diff () { + expect_code=$1 + expect_out=$2 + expect_err=$3 + command_code=$4 + trust_exit_code=$5 + shift 5 + options="$@" + + command="echo output; exit $command_code;" + desc="external diff '$command' with trustExitCode=$trust_exit_code" + with_options="${options:+ with }$options" + + test_expect_success "$desc via attribute$with_options" " + test_config diff.foo.command \"$command\" && + test_config diff.foo.trustExitCode $trust_exit_code && + echo \"file diff=foo\" >.gitattributes && + test_expect_code $expect_code git diff $options >out 2>err && + test_cmp $expect_out out && + test_cmp $expect_err err + " + + test_expect_success "$desc via diff.external$with_options" " + test_config diff.external \"$command\" && + test_config diff.trustExitCode $trust_exit_code && + >.gitattributes && + test_expect_code $expect_code git diff $options >out 2>err && + test_cmp $expect_out out && + test_cmp $expect_err err + " + + test_expect_success "$desc via GIT_EXTERNAL_DIFF$with_options" " + >.gitattributes && + test_expect_code $expect_code env \ + GIT_EXTERNAL_DIFF=\"$command\" \ + GIT_EXTERNAL_DIFF_TRUST_EXIT_CODE=$trust_exit_code \ + git diff $options >out 2>err && + test_cmp $expect_out out && + test_cmp $expect_err err + " +} + +test_expect_success 'setup output files' ' + : >empty && + echo output >output && + echo "fatal: external diff died, stopping at file" >error +' + +check_external_diff 0 output empty 0 off +check_external_diff 128 output error 1 off +check_external_diff 0 output empty 0 on +check_external_diff 0 output empty 1 on +check_external_diff 128 output error 2 on + +check_external_diff 1 output empty 0 off --exit-code +check_external_diff 128 output error 1 off --exit-code +check_external_diff 0 output empty 0 on --exit-code +check_external_diff 1 output empty 1 on --exit-code +check_external_diff 128 output error 2 on --exit-code + +check_external_diff 1 empty empty 0 off --quiet +check_external_diff 1 empty empty 1 off --quiet # we don't even call the program +check_external_diff 0 empty empty 0 on --quiet +check_external_diff 1 empty empty 1 on --quiet +check_external_diff 128 empty error 2 on --quiet + echo NULZbetweenZwords | perl -pe 'y/Z/\000/' > file test_expect_success 'force diff with "diff"' ' diff --git a/t/t4202-log.sh b/t/t4202-log.sh index 86c695eb0a..51f7beb59f 100755 --- a/t/t4202-log.sh +++ b/t/t4202-log.sh @@ -1237,6 +1237,30 @@ test_expect_success 'log.abbrevCommit configuration' ' test_cmp expect.whatchanged.full actual ' +test_expect_success '--abbrev-commit with core.abbrev=false' ' + git log --no-abbrev >expect && + git -c core.abbrev=false log --abbrev-commit >actual && + test_cmp expect actual +' + +test_expect_success '--abbrev-commit with --no-abbrev' ' + git log --no-abbrev >expect && + git log --abbrev-commit --no-abbrev >actual && + test_cmp expect actual +' + +test_expect_success '--abbrev-commit with core.abbrev=9000' ' + git log --no-abbrev >expect && + git -c core.abbrev=9000 log --abbrev-commit >actual && + test_cmp expect actual +' + +test_expect_success '--abbrev-commit with --abbrev=9000' ' + git log --no-abbrev >expect && + git log --abbrev-commit --abbrev=9000 >actual && + test_cmp expect actual +' + test_expect_success 'show added path under "--follow -M"' ' # This tests for a regression introduced in v1.7.2-rc0~103^2~2 test_create_repo regression && diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh index 281266f788..77e91547ea 100755 --- a/t/t5324-split-commit-graph.sh +++ b/t/t5324-split-commit-graph.sh @@ -13,7 +13,8 @@ test_expect_success 'setup repo' ' git init && git config core.commitGraph true && git config gc.writeCommitGraph false && - infodir=".git/objects/info" && + objdir=".git/objects" && + infodir="$objdir/info" && graphdir="$infodir/commit-graphs" && test_oid_cache <<-EOM shallow sha1:2132 @@ -718,4 +719,27 @@ test_expect_success 'write generation data chunk when commit-graph chain is repl ) ' +test_expect_success 'temporary graph layer is discarded upon failure' ' + git init layer-discard && + ( + cd layer-discard && + + test_commit A && + test_commit B && + + # Intentionally remove commit "A" from the object store + # so that the commit-graph machinery fails to parse the + # parents of "B". + # + # This takes place after the commit-graph machinery has + # initialized a new temporary file to store the contents + # of the new graph layer, so will allow us to ensure + # that the temporary file is discarded upon failure. + rm $objdir/$(test_oid_to_path $(git rev-parse HEAD^)) && + + test_must_fail git commit-graph write --reachable --split && + test_dir_is_empty $graphdir + ) +' + test_done diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh index cc7220b6c0..916da389b6 100755 --- a/t/t5326-multi-pack-bitmaps.sh +++ b/t/t5326-multi-pack-bitmaps.sh @@ -551,4 +551,34 @@ do ' done +test_expect_success 'remove one packfile between MIDX bitmap writes' ' + git init remove-pack-between-writes && + ( + cd remove-pack-between-writes && + + test_commit A && + test_commit B && + test_commit C && + + # Create packs with the prefix "pack-A", "pack-B", + # "pack-C" to impose a lexicographic order on these + # packs so the pack being removed is always from the + # middle. + packdir=.git/objects/pack && + A="$(echo A | git pack-objects $packdir/pack-A --revs)" && + B="$(echo B | git pack-objects $packdir/pack-B --revs)" && + C="$(echo C | git pack-objects $packdir/pack-C --revs)" && + + git multi-pack-index write --bitmap && + + cat >in <<-EOF && + pack-A-$A.idx + pack-C-$C.idx + EOF + git multi-pack-index write --bitmap --stdin-packs <in && + + git rev-list --test-bitmap HEAD + ) +' + test_done diff --git a/t/t5332-multi-pack-reuse.sh b/t/t5332-multi-pack-reuse.sh index 3c20738bce..ed823f37bc 100755 --- a/t/t5332-multi-pack-reuse.sh +++ b/t/t5332-multi-pack-reuse.sh @@ -204,4 +204,30 @@ test_expect_success 'omit delta from uninteresting base (cross pack)' ' test_pack_objects_reused_all $(($objects_nr - 1)) $packs_nr ' +test_expect_success 'non-omitted delta in MIDX preferred pack' ' + test_config pack.allowPackReuse single && + + cat >p1.objects <<-EOF && + $(git rev-parse $base) + ^$(git rev-parse $delta^) + EOF + cat >p2.objects <<-EOF && + $(git rev-parse F) + EOF + + p1="$(git pack-objects --revs $packdir/pack <p1.objects)" && + p2="$(git pack-objects --revs $packdir/pack <p2.objects)" && + + cat >in <<-EOF && + pack-$p1.idx + pack-$p2.idx + EOF + git multi-pack-index write --bitmap --stdin-packs \ + --preferred-pack=pack-$p1.pack <in && + + git show-index <$packdir/pack-$p1.idx >expect && + + test_pack_objects_reused_all $(wc -l <expect) 1 +' + test_done diff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh new file mode 100755 index 0000000000..f052f395a7 --- /dev/null +++ b/t/t5333-pseudo-merge-bitmaps.sh @@ -0,0 +1,393 @@ +#!/bin/sh + +test_description='pseudo-merge bitmaps' + +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 + +. ./test-lib.sh + +test_pseudo_merges () { + test-tool bitmap dump-pseudo-merges +} + +test_pseudo_merge_commits () { + test-tool bitmap dump-pseudo-merge-commits "$1" +} + +test_pseudo_merges_satisfied () { + test_trace2_data bitmap pseudo_merges_satisfied "$1" +} + +test_pseudo_merges_cascades () { + test_trace2_data bitmap pseudo_merges_cascades "$1" +} + +test_pseudo_merges_reused () { + test_trace2_data pack-bitmap-write building_bitmaps_pseudo_merge_reused "$1" +} + +tag_everything () { + git rev-list --all --no-object-names >in && + perl -lne ' + print "create refs/tags/" . $. . " " . $1 if /([0-9a-f]+)/ + ' <in | git update-ref --stdin +} + +test_expect_success 'setup' ' + test_commit_bulk 512 && + tag_everything +' + +test_expect_success 'bitmap traversal without pseudo-merges' ' + git repack -adb && + + git rev-list --count --all --objects >expect && + + : >trace2.txt && + GIT_TRACE2_EVENT=$PWD/trace2.txt \ + git rev-list --count --all --objects --use-bitmap-index >actual && + + test_pseudo_merges_satisfied 0 <trace2.txt && + test_pseudo_merges_cascades 0 <trace2.txt && + test_pseudo_merges >merges && + test_must_be_empty merges && + test_cmp expect actual +' + +test_expect_success 'pseudo-merges accurately represent their objects' ' + test_config bitmapPseudoMerge.test.pattern "refs/tags/" && + test_config bitmapPseudoMerge.test.maxMerges 8 && + test_config bitmapPseudoMerge.test.stableThreshold never && + + git repack -adb && + + test_pseudo_merges >merges && + test_line_count = 8 merges && + + for i in $(test_seq 0 $(($(wc -l <merges)-1))) + do + test-tool bitmap dump-pseudo-merge-commits $i >commits && + + git rev-list --objects --no-object-names --stdin <commits >expect.raw && + test-tool bitmap dump-pseudo-merge-objects $i >actual.raw && + + sort -u <expect.raw >expect && + sort -u <actual.raw >actual && + + test_cmp expect actual || return 1 + done +' + +test_expect_success 'bitmap traversal with pseudo-merges' ' + : >trace2.txt && + GIT_TRACE2_EVENT=$PWD/trace2.txt \ + git rev-list --count --all --objects --use-bitmap-index >actual && + git rev-list --count --all --objects >expect && + + test_pseudo_merges_satisfied 8 <trace2.txt && + test_pseudo_merges_cascades 1 <trace2.txt && + test_cmp expect actual +' + +test_expect_success 'stale bitmap traversal with pseudo-merges' ' + test_commit other && + + : >trace2.txt && + GIT_TRACE2_EVENT=$PWD/trace2.txt \ + git rev-list --count --all --objects --use-bitmap-index >actual && + git rev-list --count --all --objects >expect && + + test_pseudo_merges_satisfied 8 <trace2.txt && + test_pseudo_merges_cascades 1 <trace2.txt && + test_cmp expect actual +' + +test_expect_success 'bitmapPseudoMerge.sampleRate adjusts commit selection rate' ' + test_config bitmapPseudoMerge.test.pattern "refs/tags/" && + test_config bitmapPseudoMerge.test.maxMerges 1 && + test_config bitmapPseudoMerge.test.stableThreshold never && + + commits_nr=$(git rev-list --all --count) && + + for rate in 1.0 0.5 0.25 + do + git -c bitmapPseudoMerge.test.sampleRate=$rate repack -adb && + + test_pseudo_merges >merges && + test_line_count = 1 merges && + test_pseudo_merge_commits 0 >commits && + + test-tool bitmap list-commits >bitmaps && + bitmaps_nr="$(wc -l <bitmaps)" && + + perl -MPOSIX -e "print ceil(\$ARGV[0]*(\$ARGV[1]-\$ARGV[2]))" \ + "$rate" "$commits_nr" "$bitmaps_nr" >expect && + + test $(cat expect) -eq $(wc -l <commits) || return 1 + done +' + +test_expect_success 'bitmapPseudoMerge.threshold excludes newer commits' ' + git init pseudo-merge-threshold && + ( + cd pseudo-merge-threshold && + + new="1672549200" && # 2023-01-01 + old="1641013200" && # 2022-01-01 + + GIT_COMMITTER_DATE="$new +0000" && + export GIT_COMMITTER_DATE && + test_commit_bulk --message="new" --notick 128 && + + GIT_COMMITTER_DATE="$old +0000" && + export GIT_COMMITTER_DATE && + test_commit_bulk --message="old" --notick 128 && + + tag_everything && + + git \ + -c bitmapPseudoMerge.test.pattern="refs/tags/" \ + -c bitmapPseudoMerge.test.maxMerges=1 \ + -c bitmapPseudoMerge.test.threshold=$(($new - 1)) \ + -c bitmapPseudoMerge.test.stableThreshold=never \ + repack -adb && + + test_pseudo_merges >merges && + test_line_count = 1 merges && + + test_pseudo_merge_commits 0 >oids && + git cat-file --batch <oids >commits && + + test $(wc -l <oids) = $(grep -c "^committer.*$old +0000$" commits) + ) +' + +test_expect_success 'bitmapPseudoMerge.stableThreshold creates stable groups' ' + ( + cd pseudo-merge-threshold && + + new="1672549200" && # 2023-01-01 + mid="1654059600" && # 2022-06-01 + old="1641013200" && # 2022-01-01 + + GIT_COMMITTER_DATE="$mid +0000" && + export GIT_COMMITTER_DATE && + test_commit_bulk --message="mid" --notick 128 && + + git for-each-ref --format="delete %(refname)" refs/tags >in && + git update-ref --stdin <in && + + tag_everything && + + git \ + -c bitmapPseudoMerge.test.pattern="refs/tags/" \ + -c bitmapPseudoMerge.test.maxMerges=1 \ + -c bitmapPseudoMerge.test.threshold=$(($new - 1)) \ + -c bitmapPseudoMerge.test.stableThreshold=$(($mid - 1)) \ + -c bitmapPseudoMerge.test.stableSize=10 \ + repack -adb && + + test_pseudo_merges >merges && + merges_nr="$(wc -l <merges)" && + + for i in $(test_seq $(($merges_nr - 1))) + do + test_pseudo_merge_commits 0 >oids && + git cat-file --batch <oids >commits && + + expect="$(grep -c "^committer.*$old +0000$" commits)" && + actual="$(wc -l <oids)" && + + test $expect = $actual || return 1 + done && + + test_pseudo_merge_commits $(($merges_nr - 1)) >oids && + git cat-file --batch <oids >commits && + test $(wc -l <oids) = $(grep -c "^committer.*$mid +0000$" commits) + ) +' + +test_expect_success 'out of order thresholds are rejected' ' + test_must_fail git \ + -c bitmapPseudoMerge.test.pattern="refs/*" \ + -c bitmapPseudoMerge.test.threshold=1.month.ago \ + -c bitmapPseudoMerge.test.stableThreshold=1.week.ago \ + repack -adb 2>err && + + cat >expect <<-EOF && + fatal: pseudo-merge group ${SQ}test${SQ} has unstable threshold before stable one + EOF + + test_cmp expect err +' + +test_expect_success 'pseudo-merge pattern with capture groups' ' + git init pseudo-merge-captures && + ( + cd pseudo-merge-captures && + + test_commit_bulk 128 && + tag_everything && + + for r in $(test_seq 8) + do + test_commit_bulk 16 && + + git rev-list HEAD~16.. >in && + + perl -lne "print \"create refs/remotes/$r/tags/\$. \$_\"" <in | + git update-ref --stdin || return 1 + done && + + git \ + -c bitmapPseudoMerge.tags.pattern="refs/remotes/([0-9]+)/tags/" \ + -c bitmapPseudoMerge.tags.maxMerges=1 \ + repack -adb && + + git for-each-ref --format="%(objectname) %(refname)" >refs && + + test_pseudo_merges >merges && + for m in $(test_seq 0 $(($(wc -l <merges) - 1))) + do + test_pseudo_merge_commits $m >oids && + grep -f oids refs | + perl -lne "print \$1 if /refs\/remotes\/([0-9]+)/" | + sort -u || return 1 + done >remotes && + + test $(wc -l <remotes) -eq $(sort -u <remotes | wc -l) + ) +' + +test_expect_success 'pseudo-merge overlap setup' ' + git init pseudo-merge-overlap && + ( + cd pseudo-merge-overlap && + + test_commit_bulk 256 && + tag_everything && + + git \ + -c bitmapPseudoMerge.all.pattern="refs/" \ + -c bitmapPseudoMerge.all.maxMerges=1 \ + -c bitmapPseudoMerge.all.stableThreshold=never \ + -c bitmapPseudoMerge.tags.pattern="refs/tags/" \ + -c bitmapPseudoMerge.tags.maxMerges=1 \ + -c bitmapPseudoMerge.tags.stableThreshold=never \ + repack -adb + ) +' + +test_expect_success 'pseudo-merge overlap generates overlapping groups' ' + ( + cd pseudo-merge-overlap && + + test_pseudo_merges >merges && + test_line_count = 2 merges && + + test_pseudo_merge_commits 0 >commits-0.raw && + test_pseudo_merge_commits 1 >commits-1.raw && + + sort commits-0.raw >commits-0 && + sort commits-1.raw >commits-1 && + + comm -12 commits-0 commits-1 >overlap && + + test_line_count -gt 0 overlap + ) +' + +test_expect_success 'pseudo-merge overlap traversal' ' + ( + cd pseudo-merge-overlap && + + : >trace2.txt && + GIT_TRACE2_EVENT=$PWD/trace2.txt \ + git rev-list --count --all --objects --use-bitmap-index >actual && + git rev-list --count --all --objects >expect && + + test_pseudo_merges_satisfied 2 <trace2.txt && + test_pseudo_merges_cascades 1 <trace2.txt && + test_cmp expect actual + ) +' + +test_expect_success 'pseudo-merge overlap stale traversal' ' + ( + cd pseudo-merge-overlap && + + test_commit other && + + : >trace2.txt && + GIT_TRACE2_EVENT=$PWD/trace2.txt \ + git rev-list --count --all --objects --use-bitmap-index >actual && + git rev-list --count --all --objects >expect && + + test_pseudo_merges_satisfied 2 <trace2.txt && + test_pseudo_merges_cascades 1 <trace2.txt && + test_cmp expect actual + ) +' + +test_expect_success 'pseudo-merge reuse' ' + git init pseudo-merge-reuse && + ( + cd pseudo-merge-reuse && + + stable="1641013200" && # 2022-01-01 + unstable="1672549200" && # 2023-01-01 + + GIT_COMMITTER_DATE="$stable +0000" && + export GIT_COMMITTER_DATE && + test_commit_bulk --notick 128 && + GIT_COMMITTER_DATE="$unstable +0000" && + export GIT_COMMITTER_DATE && + test_commit_bulk --notick 128 && + + tag_everything && + + git \ + -c bitmapPseudoMerge.test.pattern="refs/tags/" \ + -c bitmapPseudoMerge.test.maxMerges=1 \ + -c bitmapPseudoMerge.test.threshold=now \ + -c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \ + -c bitmapPseudoMerge.test.stableSize=512 \ + repack -adb && + + test_pseudo_merges >merges && + test_line_count = 2 merges && + + test_pseudo_merge_commits 0 >stable-oids.before && + test_pseudo_merge_commits 1 >unstable-oids.before && + + : >trace2.txt && + GIT_TRACE2_EVENT=$PWD/trace2.txt git \ + -c bitmapPseudoMerge.test.pattern="refs/tags/" \ + -c bitmapPseudoMerge.test.maxMerges=2 \ + -c bitmapPseudoMerge.test.threshold=now \ + -c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \ + -c bitmapPseudoMerge.test.stableSize=512 \ + repack -adb && + + test_pseudo_merges_reused 1 <trace2.txt && + + test_pseudo_merges >merges && + test_line_count = 3 merges && + + test_pseudo_merge_commits 0 >stable-oids.after && + for i in 1 2 + do + test_pseudo_merge_commits $i || return 1 + done >unstable-oids.after && + + sort -u <stable-oids.before >expect && + sort -u <stable-oids.after >actual && + test_cmp expect actual && + + sort -u <unstable-oids.before >expect && + sort -u <unstable-oids.after >actual && + test_cmp expect actual + ) +' + +test_done diff --git a/t/t5512-ls-remote.sh b/t/t5512-ls-remote.sh index 5dbe107ce8..42e77eb5a9 100755 --- a/t/t5512-ls-remote.sh +++ b/t/t5512-ls-remote.sh @@ -47,6 +47,7 @@ test_expect_success setup ' git show-ref -d >refs && sed -e "s/ / /" refs >>expected.all && + grep refs/heads/ expected.all >expected.branches && git remote add self "$(pwd)/.git" && git remote add self2 "." ' @@ -71,6 +72,27 @@ test_expect_success 'ls-remote self' ' test_cmp expected.all actual ' +test_expect_success 'ls-remote --branches self' ' + git ls-remote --branches self >actual && + test_cmp expected.branches actual && + git ls-remote -b self >actual && + test_cmp expected.branches actual +' + +test_expect_success 'ls-remote -h is deprecated w/o warning' ' + git ls-remote -h self >actual 2>warning && + test_cmp expected.branches actual && + test_grep ! deprecated warning +' + +test_expect_success 'ls-remote --heads is deprecated and hidden w/o warning' ' + test_expect_code 129 git ls-remote -h >short-help && + test_grep ! -e --head short-help && + git ls-remote --heads self >actual 2>warning && + test_cmp expected.branches actual && + test_grep ! deprecated warning +' + test_expect_success 'ls-remote --sort="version:refname" --tags self' ' generate_references \ refs/tags/mark \ @@ -275,7 +297,7 @@ test_expect_success 'ls-remote with filtered symref (refname)' ' test_cmp expect actual ' -test_expect_success 'ls-remote with filtered symref (--heads)' ' +test_expect_success 'ls-remote with filtered symref (--branches)' ' git symbolic-ref refs/heads/foo refs/tags/mark && cat >expect.v2 <<-EOF && ref: refs/tags/mark refs/heads/foo @@ -283,9 +305,9 @@ test_expect_success 'ls-remote with filtered symref (--heads)' ' $rev refs/heads/main EOF grep -v "^ref: refs/tags/" <expect.v2 >expect.v0 && - git -c protocol.version=0 ls-remote --symref --heads . >actual.v0 && + git -c protocol.version=0 ls-remote --symref --branches . >actual.v0 && test_cmp expect.v0 actual.v0 && - git -c protocol.version=2 ls-remote --symref --heads . >actual.v2 && + git -c protocol.version=2 ls-remote --symref --branches . >actual.v2 && test_cmp expect.v2 actual.v2 ' @@ -335,9 +357,9 @@ test_expect_success 'ls-remote patterns work with all protocol versions' ' test_expect_success 'ls-remote prefixes work with all protocol versions' ' git for-each-ref --format="%(objectname) %(refname)" \ refs/heads/ refs/tags/ >expect && - git -c protocol.version=0 ls-remote --heads --tags . >actual.v0 && + git -c protocol.version=0 ls-remote --branches --tags . >actual.v0 && test_cmp expect actual.v0 && - git -c protocol.version=2 ls-remote --heads --tags . >actual.v2 && + git -c protocol.version=2 ls-remote --branches --tags . >actual.v2 && test_cmp expect actual.v2 ' diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index cc0b953f14..5d7ea147f1 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -46,6 +46,13 @@ test_expect_success 'output from clone' ' test $(grep Clon output | wc -l) = 1 ' +test_expect_success 'output from clone with core.abbrev does not crash' ' + rm -fr dst && + echo "Cloning into ${SQ}dst${SQ}..." >expect && + git -c core.abbrev=12 clone -n "file://$(pwd)/src" dst >actual 2>&1 && + test_cmp expect actual +' + test_expect_success 'clone does not keep pack' ' rm -fr dst && diff --git a/t/t5605-clone-local.sh b/t/t5605-clone-local.sh index a3055869bc..339d8c786f 100755 --- a/t/t5605-clone-local.sh +++ b/t/t5605-clone-local.sh @@ -163,7 +163,7 @@ test_expect_success REFFILES 'local clone from repo with corrupt refs fails grac echo a >corrupt/.git/refs/heads/topic && test_must_fail git clone corrupt working 2>err && - grep "has a null OID" err + grep "has neither a valid OID nor a target" err ' test_done diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh index 43d40175f8..1b5909d1b7 100755 --- a/t/t6500-gc.sh +++ b/t/t6500-gc.sh @@ -158,7 +158,7 @@ test_expect_success TTY 'with TTY: gc --no-quiet' ' git -c gc.writeCommitGraph=true gc --no-quiet >stdout 2>stderr && test_must_be_empty stdout && test_grep "Enumerating objects" stderr && - test_grep "Computing commit graph generation numbers" stderr + test_grep "Computing commit graph generation numbers: 100% (4/4), done." stderr ' test_expect_success 'gc --quiet' ' diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh index 862d80c974..427b375b39 100644 --- a/t/test-lib-functions.sh +++ b/t/test-lib-functions.sh @@ -458,6 +458,7 @@ test_commit_bulk () { indir=. ref=HEAD n=1 + notick= message='commit %s' filename='%s.t' contents='content %s' @@ -488,6 +489,9 @@ test_commit_bulk () { filename="${1#--*=}-%s.t" contents="${1#--*=} %s" ;; + --notick) + notick=yes + ;; -*) BUG "invalid test_commit_bulk option: $1" ;; @@ -507,7 +511,10 @@ test_commit_bulk () { while test "$total" -gt 0 do - test_tick && + if test -z "$notick" + then + test_tick + fi && echo "commit $ref" printf 'author %s <%s> %s\n' \ "$GIT_AUTHOR_NAME" \ diff --git a/t/unit-tests/lib-oid.c b/t/unit-tests/lib-oid.c new file mode 100644 index 0000000000..37105f0a8f --- /dev/null +++ b/t/unit-tests/lib-oid.c @@ -0,0 +1,52 @@ +#include "test-lib.h" +#include "lib-oid.h" +#include "strbuf.h" +#include "hex.h" + +static int init_hash_algo(void) +{ + static int algo = -1; + + if (algo < 0) { + const char *algo_name = getenv("GIT_TEST_DEFAULT_HASH"); + algo = algo_name ? hash_algo_by_name(algo_name) : GIT_HASH_SHA1; + + if (!check(algo != GIT_HASH_UNKNOWN)) + test_msg("BUG: invalid GIT_TEST_DEFAULT_HASH value ('%s')", + algo_name); + } + return algo; +} + +static int get_oid_arbitrary_hex_algop(const char *hex, struct object_id *oid, + const struct git_hash_algo *algop) +{ + int ret; + size_t sz = strlen(hex); + struct strbuf buf = STRBUF_INIT; + + if (!check(sz <= algop->hexsz)) { + test_msg("BUG: hex string (%s) bigger than maximum allowed (%lu)", + hex, (unsigned long)algop->hexsz); + return -1; + } + + strbuf_add(&buf, hex, sz); + strbuf_addchars(&buf, '0', algop->hexsz - sz); + + ret = get_oid_hex_algop(buf.buf, oid, algop); + if (!check_int(ret, ==, 0)) + test_msg("BUG: invalid hex input (%s) provided", hex); + + strbuf_release(&buf); + return ret; +} + +int get_oid_arbitrary_hex(const char *hex, struct object_id *oid) +{ + int hash_algo = init_hash_algo(); + + if (!check_int(hash_algo, !=, GIT_HASH_UNKNOWN)) + return -1; + return get_oid_arbitrary_hex_algop(hex, oid, &hash_algos[hash_algo]); +} diff --git a/t/unit-tests/lib-oid.h b/t/unit-tests/lib-oid.h new file mode 100644 index 0000000000..bfde639190 --- /dev/null +++ b/t/unit-tests/lib-oid.h @@ -0,0 +1,17 @@ +#ifndef LIB_OID_H +#define LIB_OID_H + +#include "hash-ll.h" + +/* + * Convert arbitrary hex string to object_id. + * For example, passing "abc12" will generate + * "abc1200000000000000000000000000000000000" hex of length 40 for SHA-1 and + * create object_id with that. + * WARNING: passing a string of length more than the hexsz of respective hash + * algo is not allowed. The hash algo is decided based on GIT_TEST_DEFAULT_HASH + * environment variable. + */ +int get_oid_arbitrary_hex(const char *s, struct object_id *oid); + +#endif /* LIB_OID_H */ diff --git a/t/unit-tests/t-oidtree.c b/t/unit-tests/t-oidtree.c new file mode 100644 index 0000000000..a38754b066 --- /dev/null +++ b/t/unit-tests/t-oidtree.c @@ -0,0 +1,122 @@ +#include "test-lib.h" +#include "lib-oid.h" +#include "oidtree.h" +#include "hash.h" +#include "hex.h" +#include "strvec.h" + +#define FILL_TREE(tree, ...) \ + do { \ + const char *hexes[] = { __VA_ARGS__ }; \ + if (fill_tree_loc(tree, hexes, ARRAY_SIZE(hexes))) \ + return; \ + } while (0) + +static int fill_tree_loc(struct oidtree *ot, const char *hexes[], size_t n) +{ + for (size_t i = 0; i < n; i++) { + struct object_id oid; + if (!check_int(get_oid_arbitrary_hex(hexes[i], &oid), ==, 0)) + return -1; + oidtree_insert(ot, &oid); + } + return 0; +} + +static void check_contains(struct oidtree *ot, const char *hex, int expected) +{ + struct object_id oid; + + if (!check_int(get_oid_arbitrary_hex(hex, &oid), ==, 0)) + return; + if (!check_int(oidtree_contains(ot, &oid), ==, expected)) + test_msg("oid: %s", oid_to_hex(&oid)); +} + +struct expected_hex_iter { + size_t i; + struct strvec expected_hexes; + const char *query; +}; + +static enum cb_next check_each_cb(const struct object_id *oid, void *data) +{ + struct expected_hex_iter *hex_iter = data; + struct object_id expected; + + if (!check_int(hex_iter->i, <, hex_iter->expected_hexes.nr)) { + test_msg("error: extraneous callback for query: ('%s'), object_id: ('%s')", + hex_iter->query, oid_to_hex(oid)); + return CB_BREAK; + } + + if (!check_int(get_oid_arbitrary_hex(hex_iter->expected_hexes.v[hex_iter->i], + &expected), ==, 0)) + ; /* the data is bogus and cannot be used */ + else if (!check(oideq(oid, &expected))) + test_msg("expected: %s\n got: %s\n query: %s", + oid_to_hex(&expected), oid_to_hex(oid), hex_iter->query); + + hex_iter->i += 1; + return CB_CONTINUE; +} + +LAST_ARG_MUST_BE_NULL +static void check_each(struct oidtree *ot, const char *query, ...) +{ + struct object_id oid; + struct expected_hex_iter hex_iter = { .expected_hexes = STRVEC_INIT, + .query = query }; + const char *arg; + va_list hex_args; + + va_start(hex_args, query); + while ((arg = va_arg(hex_args, const char *))) + strvec_push(&hex_iter.expected_hexes, arg); + va_end(hex_args); + + if (!check_int(get_oid_arbitrary_hex(query, &oid), ==, 0)) + return; + oidtree_each(ot, &oid, strlen(query), check_each_cb, &hex_iter); + + if (!check_int(hex_iter.i, ==, hex_iter.expected_hexes.nr)) + test_msg("error: could not find some 'object_id's for query ('%s')", query); + strvec_clear(&hex_iter.expected_hexes); +} + +static void setup(void (*f)(struct oidtree *ot)) +{ + struct oidtree ot; + + oidtree_init(&ot); + f(&ot); + oidtree_clear(&ot); +} + +static void t_contains(struct oidtree *ot) +{ + FILL_TREE(ot, "444", "1", "2", "3", "4", "5", "a", "b", "c", "d", "e"); + check_contains(ot, "44", 0); + check_contains(ot, "441", 0); + check_contains(ot, "440", 0); + check_contains(ot, "444", 1); + check_contains(ot, "4440", 1); + check_contains(ot, "4444", 0); +} + +static void t_each(struct oidtree *ot) +{ + FILL_TREE(ot, "f", "9", "8", "123", "321", "320", "a", "b", "c", "d", "e"); + check_each(ot, "12300", "123", NULL); + check_each(ot, "3211", NULL); /* should not reach callback */ + check_each(ot, "3210", "321", NULL); + check_each(ot, "32100", "321", NULL); + check_each(ot, "32", "320", "321", NULL); +} + +int cmd_main(int argc UNUSED, const char **argv UNUSED) +{ + TEST(setup(t_contains), "oidtree insert and contains works"); + TEST(setup(t_each), "oidtree each works"); + return test_done(); +} diff --git a/userdiff.c b/userdiff.c index 371032a413..c4ebb9ff73 100644 --- a/userdiff.c +++ b/userdiff.c @@ -333,7 +333,7 @@ PATTERNS("scheme", "|([^][)(}{[ \t])+"), PATTERNS("tex", "^(\\\\((sub)*section|chapter|part)\\*{0,1}\\{.*)$", "\\\\[a-zA-Z@]+|\\\\.|([a-zA-Z0-9]|[^\x01-\x7f])+"), -{ "default", NULL, NULL, -1, { NULL, 0 } }, +{ .name = "default", .binary = -1 }, }; #undef PATTERNS #undef IPATTERN @@ -445,7 +445,11 @@ int userdiff_config(const char *k, const char *v) if (!strcmp(type, "binary")) return parse_tristate(&drv->binary, k, v); if (!strcmp(type, "command")) - return git_config_string((char **) &drv->external, k, v); + return git_config_string((char **) &drv->external.cmd, k, v); + if (!strcmp(type, "trustexitcode")) { + drv->external.trust_exit_code = git_config_bool(k, v); + return 0; + } if (!strcmp(type, "textconv")) return git_config_string((char **) &drv->textconv, k, v); if (!strcmp(type, "cachetextconv")) diff --git a/userdiff.h b/userdiff.h index d726804c3e..7565930337 100644 --- a/userdiff.h +++ b/userdiff.h @@ -11,9 +11,14 @@ struct userdiff_funcname { int cflags; }; +struct external_diff { + char *cmd; + unsigned trust_exit_code:1; +}; + struct userdiff_driver { const char *name; - const char *external; + struct external_diff external; const char *algorithm; int binary; struct userdiff_funcname funcname; diff --git a/worktree.h b/worktree.h index 7cc6d90e66..11279d0c8f 100644 --- a/worktree.h +++ b/worktree.h @@ -178,14 +178,6 @@ int is_worktree_being_rebased(const struct worktree *wt, const char *target); int is_worktree_being_bisected(const struct worktree *wt, const char *target); /* - * Similar to git_path() but can produce paths for a specified - * worktree instead of current one - */ -const char *worktree_git_path(const struct worktree *wt, - const char *fmt, ...) - __attribute__((format (printf, 2, 3))); - -/* * Return a refname suitable for access from the current ref store. */ void strbuf_worktree_ref(const struct worktree *wt, |