From a45e390ad6ad1f26967a40e7771777f70a77d719 Mon Sep 17 00:00:00 2001 From: Greg Hurrell Date: Wed, 9 Jun 2021 21:28:06 +0200 Subject: [PATCH 001/762] gitweb: use HEAD as secondary sort key in git_get_heads_list() The "heads" section on the gitweb summary page shows heads in `-committerdate` order (ie. the most recently-modified ones at the top), tie-breaking equal-dated refs using the implicit `refname` sort fallback. This recency-based ordering appears in multiple places in the UI, such as the project listing, the tags list, and even the shortlog and log views. Given two equal-dated refs, however, sorting the `HEAD` ref before the non-`HEAD` ref provides more useful signal than merely sorting by refname. For example, say we had "master" and "trunk" both pointing at the same commit but "trunk" was `HEAD`, sorting "trunk" first helps communicate its special status as the default branch that you'll check out if you clone the repo. Add `-HEAD` as a secondary sort key to the `git for-each-ref` call in `git_get_heads_list()` to provide the desired behavior. The most recently committed refs will appear first, but `HEAD`-ness will be used as a tie-breaker. Note that `refname` is the implicit fallback sort key, which means that two same-dated non-`HEAD` refs will continue to be sorted in lexicographical order, as they are today. Signed-off-by: Greg Hurrell Signed-off-by: Junio C Hamano --- gitweb/gitweb.perl | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl index 0959a782ec..092e707ee1 100755 --- a/gitweb/gitweb.perl +++ b/gitweb/gitweb.perl @@ -3779,7 +3779,8 @@ sub git_get_heads_list { my @headslist; open my $fd, '-|', git_cmd(), 'for-each-ref', - ($limit ? '--count='.($limit+1) : ()), '--sort=-committerdate', + ($limit ? '--count='.($limit+1) : ()), + '--sort=-HEAD', '--sort=-committerdate', '--format=%(objectname) %(refname) %(subject)%00%(committer)', @patterns or return; From fb20d4b1268d97646ae24f07661892cf6da64c31 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Mon, 21 Jun 2021 17:03:37 +0200 Subject: [PATCH 002/762] pack-objects tests: cover blindspots in stdin handling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cover blindspots in the testing of stdin handling, including the "!len" condition added in b5d97e6b0a0 (pack-objects: run rev-list equivalent internally., 2006-09-04). The codepath taken with --revs and read_object_list_from_stdin() acts differently in some of these common cases, let's test for those. The "--stdin --revs" test being added here stresses the combination of --stdin-packs and the revision.c --stdin argument, some of this was covered in a test added in 339bce27f4f (builtin/pack-objects.c: add '--stdin-packs' option, 2021-02-22), but let's make sure that GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS=true keeps erroring out about --stdin, and it isn't picked up by the revision.c API's handling of that option. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- t/t5300-pack-object.sh | 85 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh index 5c5e53f0be..65e991e370 100755 --- a/t/t5300-pack-object.sh +++ b/t/t5300-pack-object.sh @@ -34,6 +34,91 @@ test_expect_success 'setup' ' } >expect ' +test_expect_success 'setup pack-object in <<-EOF && + $(git -C pack-object-stdin rev-parse one) + EOF + + git -C pack-object-stdin pack-objects basic-stdin actual && + test_line_count = 1 actual && + + git -C pack-object-stdin pack-objects --revs basic-stdin-revs actual && + test_line_count = 3 actual +' + +test_expect_success 'pack-object in <<-EOF && + $(git -C pack-object-stdin rev-parse one) + garbage + $(git -C pack-object-stdin rev-parse two) + EOF + + sed "s/^> //g" >err.expect <<-EOF && + fatal: expected object ID, got garbage: + > garbage + + EOF + test_must_fail git -C pack-object-stdin pack-objects bad-line-stdin err.actual && + test_cmp err.expect err.actual && + + cat >err.expect <<-EOF && + fatal: bad revision '"'"'garbage'"'"' + EOF + test_must_fail git -C pack-object-stdin pack-objects --revs bad-line-stdin-revs err.actual && + test_cmp err.expect err.actual +' + +test_expect_success 'pack-object in <<-EOF && + $(git -C pack-object-stdin rev-parse one) + + $(git -C pack-object-stdin rev-parse two) + EOF + + sed -e "s/^> //g" -e "s/Z$//g" >err.expect <<-EOF && + fatal: expected object ID, got garbage: + > Z + + EOF + test_must_fail git -C pack-object-stdin pack-objects empty-line-stdin err.actual && + test_cmp err.expect err.actual && + + git -C pack-object-stdin pack-objects --revs empty-line-stdin-revs actual && + test_line_count = 3 actual +' + +test_expect_success 'pack-object in <<-EOF && + $(git -C pack-object-stdin rev-parse one) + $(git -C pack-object-stdin rev-parse two) + EOF + + # There is the "--stdin-packs is incompatible with --revs" + # test below, but we should make sure that the revision.c + # --stdin is not picked up + cat >err.expect <<-EOF && + fatal: disallowed abbreviated or ambiguous option '"'"'stdin'"'"' + EOF + test_must_fail git -C pack-object-stdin pack-objects stdin-with-stdin-option --stdin err.actual && + test_cmp err.expect err.actual && + + test_must_fail git -C pack-object-stdin pack-objects --stdin --revs stdin-with-stdin-option-revs 2>err.actual # e.g.: check_deltas stderr -gt 0 check_deltas() { From 922f8bbbf1097206969e423dd7af1bf33443765b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Tue, 29 Jun 2021 13:12:55 +0200 Subject: [PATCH 003/762] Makefile: move ".PHONY: cscope" near its target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Move the ".PHONY: cscope" rule to live alongside the "cscope" target itself, not to be all the way near the bottom where we define the "FORCE" rule. That line was last modified in 2f76919517e (MinGW: avoid collisions between "tags" and "TAGS", 2010-09-28). Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Makefile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index c3565fc0f8..4dd9711a65 100644 --- a/Makefile +++ b/Makefile @@ -2737,6 +2737,7 @@ tags: FORCE $(FIND_SOURCE_FILES) | xargs ctags -a -o tags+ && \ mv tags+ tags +.PHONY: cscope cscope: $(RM) cscope* $(FIND_SOURCE_FILES) | xargs cscope -b @@ -3245,7 +3246,7 @@ endif .PHONY: all install profile-clean cocciclean clean strip .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell -.PHONY: FORCE cscope +.PHONY: FORCE ### Check documentation # From 033395be3287170658f0a36fa406dea9def7b657 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Tue, 29 Jun 2021 13:12:56 +0200 Subject: [PATCH 004/762] Makefile: add QUIET_GEN to "cscope" target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Don't show the very verbose $(FIND_SOURCE_FILES) command on every "make cscope" invocation. See my recent 3c80fcb591 (Makefile: add QUIET_GEN to "tags" and "TAGS" targets, 2021-03-28) for the same fix for the other adjacent targets. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 4dd9711a65..faa8900097 100644 --- a/Makefile +++ b/Makefile @@ -2739,7 +2739,7 @@ tags: FORCE .PHONY: cscope cscope: - $(RM) cscope* + $(QUIET_GEN)$(RM) cscope* && \ $(FIND_SOURCE_FILES) | xargs cscope -b ### Detect prefix changes From 561fa03529202a36e0d77548fdcb5d81422c1865 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Fri, 9 Jul 2021 12:13:48 +0200 Subject: [PATCH 005/762] pack-objects: fix segfault in --stdin-packs option MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix a segfault in the --stdin-packs option added in 339bce27f4f (builtin/pack-objects.c: add '--stdin-packs' option, 2021-02-22). The read_packs_list_from_stdin() function didn't check that the lines it was reading were valid packs, and thus when doing the QSORT() with pack_mtime_cmp() we'd have a NULL "util" field. The "util" field is used to associate the names of included/excluded packs with the packed_git structs they correspond to. The logic error was in assuming that we could iterate all packs and annotate the excluded and included packs we got, as opposed to checking the lines we got on stdin. There was a check for excluded packs, but included packs were simply assumed to be valid. As noted in the test we'll not report the first bad line, but whatever line sorted first according to the string-list.c API. In this case I think that's fine. There was further discussion of alternate approaches in [1]. Even though we're being lazy let's assert the line we do expect to get in the test, since whoever changes this code in the future might miss this case, and would want to update the test and comments. 1. http://lore.kernel.org/git/YND3h2l10PlnSNGJ@nand.local Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- builtin/pack-objects.c | 23 ++++++++++++++++++++--- t/t5300-pack-object.sh | 19 +++++++++++++++++++ 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index de00adbb9e..df49f656b9 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3311,9 +3311,26 @@ static void read_packs_list_from_stdin(void) } /* - * First handle all of the excluded packs, marking them as kept in-core - * so that later calls to add_object_entry() discards any objects that - * are also found in excluded packs. + * Arguments we got on stdin may not even be packs. First + * check that to avoid segfaulting later on in + * e.g. pack_mtime_cmp(), excluded packs are handled below. + * + * Since we first parsed our STDIN and then sorted the input + * lines the pack we error on will be whatever line happens to + * sort first. This is lazy, it's enough that we report one + * bad case here, we don't need to report the first/last one, + * or all of them. + */ + for_each_string_list_item(item, &include_packs) { + struct packed_git *p = item->util; + if (!p) + die(_("could not find pack '%s'"), item->string); + } + + /* + * Then, handle all of the excluded packs, marking them as + * kept in-core so that later calls to add_object_entry() + * discards any objects that are also found in excluded packs. */ for_each_string_list_item(item, &exclude_packs) { struct packed_git *p = item->util; diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh index 65e991e370..e13a884207 100755 --- a/t/t5300-pack-object.sh +++ b/t/t5300-pack-object.sh @@ -119,6 +119,25 @@ test_expect_success 'pack-object in <<-EOF && + $(git -C pack-object-stdin rev-parse one) + $(git -C pack-object-stdin rev-parse two) + EOF + + # That we get "two" and not "one" has to do with OID + # ordering. It happens to be the same here under SHA-1 and + # SHA-256. See commentary in pack-objects.c + cat >err.expect <<-EOF && + fatal: could not find pack '"'"'$(git -C pack-object-stdin rev-parse two)'"'"' + EOF + test_must_fail git \ + -C pack-object-stdin \ + pack-objects stdin-with-stdin-option --stdin-packs \ + err.actual && + test_cmp err.expect err.actual +' + # usage: check_deltas # e.g.: check_deltas stderr -gt 0 check_deltas() { From 74fab8ff54e6e6a30efa254b8b5322764d119386 Mon Sep 17 00:00:00 2001 From: Jonathan Tan Date: Thu, 15 Jul 2021 10:44:30 -0700 Subject: [PATCH 006/762] send-pack: fix push.negotiate with remote helper Commit 477673d6f3 ("send-pack: support push negotiation", 2021-05-05) introduced the push.negotiate config variable and included a test. The test only covered pushing without a remote helper, so the fact that pushing with a remote helper doesn't work went unnoticed. This is ultimately caused by the "url" field not being set in the args struct. This field being unset probably went unnoticed because besides push negotiation, this field is only used to generate a "pushee" line in a push cert (and if not given, no such line is generated). Therefore, set this field. Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano --- builtin/send-pack.c | 1 + t/t5549-fetch-push-http.sh | 71 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+) create mode 100755 t/t5549-fetch-push-http.sh diff --git a/builtin/send-pack.c b/builtin/send-pack.c index a7e01667b0..729dea1d25 100644 --- a/builtin/send-pack.c +++ b/builtin/send-pack.c @@ -230,6 +230,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix) args.atomic = atomic; args.stateless_rpc = stateless_rpc; args.push_options = push_options.nr ? &push_options : NULL; + args.url = dest; if (from_stdin) { if (args.stateless_rpc) { diff --git a/t/t5549-fetch-push-http.sh b/t/t5549-fetch-push-http.sh new file mode 100755 index 0000000000..f50d584881 --- /dev/null +++ b/t/t5549-fetch-push-http.sh @@ -0,0 +1,71 @@ +#!/bin/sh + +test_description='fetch/push functionality using the HTTP protocol' + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh +. "$TEST_DIRECTORY"/lib-httpd.sh +start_httpd + +SERVER="$HTTPD_DOCUMENT_ROOT_PATH/server" +URI="$HTTPD_URL/smart/server" + +grep_wrote () { + object_count=$1 + file_name=$2 + grep 'write_pack_file/wrote.*"value":"'$1'"' $2 +} + +setup_client_and_server () { + git init client && + test_when_finished 'rm -rf client' && + test_commit -C client first_commit && + test_commit -C client second_commit && + + git init "$SERVER" && + test_when_finished 'rm -rf "$SERVER"' && + test_config -C "$SERVER" http.receivepack true && + git -C client push "$URI" first_commit:refs/remotes/origin/first_commit && + git -C "$SERVER" config receive.hideRefs refs/remotes/origin/first_commit +} + +test_expect_success 'push without negotiation (for comparing object counts with the next test)' ' + setup_client_and_server && + + GIT_TRACE2_EVENT="$(pwd)/event" git -C client -c protocol.version=2 \ + push "$URI" refs/heads/main:refs/remotes/origin/main && + test_when_finished "rm -f event" && + grep_wrote 6 event # 2 commits, 2 trees, 2 blobs +' + +test_expect_success 'push with negotiation' ' + setup_client_and_server && + + GIT_TRACE2_EVENT="$(pwd)/event" git -C client -c protocol.version=2 -c push.negotiate=1 \ + push "$URI" refs/heads/main:refs/remotes/origin/main && + test_when_finished "rm -f event" && + grep_wrote 3 event # 1 commit, 1 tree, 1 blob +' + +test_expect_success 'push with negotiation proceeds anyway even if negotiation fails' ' + setup_client_and_server && + + # Use protocol v0 to make negotiation fail (because protocol v0 does + # not support the "wait-for-done" capability, which is required for + # push negotiation) + GIT_TEST_PROTOCOL_VERSION=0 GIT_TRACE2_EVENT="$(pwd)/event" git -C client -c push.negotiate=1 \ + push "$URI" refs/heads/main:refs/remotes/origin/main 2>err && + test_when_finished "rm -f event" && + grep_wrote 6 event && # 2 commits, 2 trees, 2 blobs + + cat >warning-expect <<-EOF && + warning: --negotiate-only requires protocol v2 + warning: push negotiation failed; proceeding anyway with push +EOF + grep warning: err >warning-actual && + test_cmp warning-expect warning-actual +' + +test_done From 54a03bc7d9a7f264d511d88166afe8da7f75e90a Mon Sep 17 00:00:00 2001 From: Jonathan Tan Date: Thu, 15 Jul 2021 10:44:31 -0700 Subject: [PATCH 007/762] send-pack: fix push nego. when remote has refs Commit 477673d6f3 ("send-pack: support push negotiation", 2021-05-05) did not test the case in which a remote advertises at least one ref. In such a case, "remote_refs" in get_commons_through_negotiation() in send-pack.c would also contain those refs with a zero ref->new_oid (in addition to the refs being pushed with a nonzero ref->new_oid). Passing them as negotiation tips to "git fetch" causes an error, so filter them out. (The exact error that would happen in "git fetch" in this case is a segmentation fault, which is unwanted. This will be fixed in the subsequent commit.) Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano --- send-pack.c | 6 ++++-- t/t5516-fetch-push.sh | 4 +++- t/t5549-fetch-push-http.sh | 1 + 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/send-pack.c b/send-pack.c index 9cb9f71650..85945becf0 100644 --- a/send-pack.c +++ b/send-pack.c @@ -425,8 +425,10 @@ static void get_commons_through_negotiation(const char *url, child.no_stdin = 1; child.out = -1; strvec_pushl(&child.args, "fetch", "--negotiate-only", NULL); - for (ref = remote_refs; ref; ref = ref->next) - strvec_pushf(&child.args, "--negotiation-tip=%s", oid_to_hex(&ref->new_oid)); + for (ref = remote_refs; ref; ref = ref->next) { + if (!is_null_oid(&ref->new_oid)) + strvec_pushf(&child.args, "--negotiation-tip=%s", oid_to_hex(&ref->new_oid)); + } strvec_push(&child.args, url); if (start_command(&child)) diff --git a/t/t5516-fetch-push.sh b/t/t5516-fetch-push.sh index 0916f76302..4db8edd9c8 100755 --- a/t/t5516-fetch-push.sh +++ b/t/t5516-fetch-push.sh @@ -201,6 +201,7 @@ test_expect_success 'push with negotiation' ' # Without negotiation mk_empty testrepo && git push testrepo $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C testrepo unrelated_commit && git -C testrepo config receive.hideRefs refs/remotes/origin/first_commit && echo now pushing without negotiation && GIT_TRACE2_EVENT="$(pwd)/event" git -c protocol.version=2 push testrepo refs/heads/main:refs/remotes/origin/main && @@ -210,6 +211,7 @@ test_expect_success 'push with negotiation' ' rm event && mk_empty testrepo && git push testrepo $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C testrepo unrelated_commit && git -C testrepo config receive.hideRefs refs/remotes/origin/first_commit && GIT_TRACE2_EVENT="$(pwd)/event" git -c protocol.version=2 -c push.negotiate=1 push testrepo refs/heads/main:refs/remotes/origin/main && grep_wrote 2 event # 1 commit, 1 tree @@ -219,6 +221,7 @@ test_expect_success 'push with negotiation proceeds anyway even if negotiation f rm event && mk_empty testrepo && git push testrepo $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C testrepo unrelated_commit && git -C testrepo config receive.hideRefs refs/remotes/origin/first_commit && GIT_TEST_PROTOCOL_VERSION=0 GIT_TRACE2_EVENT="$(pwd)/event" \ git -c push.negotiate=1 push testrepo refs/heads/main:refs/remotes/origin/main 2>err && @@ -1767,5 +1770,4 @@ test_expect_success 'denyCurrentBranch and worktrees' ' git -C cloned push origin HEAD:new-wt && test_must_fail git -C cloned push --delete origin new-wt ' - test_done diff --git a/t/t5549-fetch-push-http.sh b/t/t5549-fetch-push-http.sh index f50d584881..2cdebcb735 100755 --- a/t/t5549-fetch-push-http.sh +++ b/t/t5549-fetch-push-http.sh @@ -27,6 +27,7 @@ setup_client_and_server () { git init "$SERVER" && test_when_finished 'rm -rf "$SERVER"' && test_config -C "$SERVER" http.receivepack true && + test_commit -C "$SERVER" unrelated_commit && git -C client push "$URI" first_commit:refs/remotes/origin/first_commit && git -C "$SERVER" config receive.hideRefs refs/remotes/origin/first_commit } From 82823118b9cd397d6b626cc86a0e555069ea8253 Mon Sep 17 00:00:00 2001 From: Jonathan Tan Date: Thu, 15 Jul 2021 10:44:32 -0700 Subject: [PATCH 008/762] fetch: die on invalid --negotiation-tip hash If a full hexadecimal hash is given as a --negotiation-tip to "git fetch", and that hash does not correspond to an object, "git fetch" will segfault if --negotiate-only is given and will silently ignore that hash otherwise. Make these cases fatal errors, just like the case when an invalid ref name or abbreviated hash is given. While at it, mark the error messages as translatable. Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano --- builtin/fetch.c | 4 +++- t/t5510-fetch.sh | 13 +++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index dfde96a435..0aeb043840 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1428,7 +1428,9 @@ static void add_negotiation_tips(struct git_transport_options *smart_options) if (!has_glob_specials(s)) { struct object_id oid; if (get_oid(s, &oid)) - die("%s is not a valid object", s); + die(_("%s is not a valid object"), s); + if (!has_object(the_repository, &oid, 0)) + die(_("the object %s does not exist"), s); oid_array_append(oids, &oid); continue; } diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh index e83b2a6506..a0faf0dd94 100755 --- a/t/t5510-fetch.sh +++ b/t/t5510-fetch.sh @@ -1214,6 +1214,19 @@ test_expect_success '--negotiation-tip understands abbreviated SHA-1' ' check_negotiation_tip ' +test_expect_success '--negotiation-tip rejects missing OIDs' ' + setup_negotiation_tip server server 0 && + test_must_fail git -C client fetch \ + --negotiation-tip=alpha_1 \ + --negotiation-tip=$(test_oid zero) \ + origin alpha_s beta_s 2>err && + cat >fatal-expect <<-EOF && + fatal: the object $(test_oid zero) does not exist +EOF + grep fatal: err >fatal-actual && + test_cmp fatal-expect fatal-actual +' + . "$TEST_DIRECTORY"/lib-httpd.sh start_httpd From be19c5ca3e28cb16d55623de2e5aa1d45bd5b5ba Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Wed, 21 Jul 2021 01:42:17 +0000 Subject: [PATCH 009/762] t7601: test interaction of merge/rebase/fast-forward flags and options The interaction of rebase and merge flags and options was not well tested. Add several tests to check for correct behavior from the following rules: * --ff-only vs. --[no-]rebase (and the related pull.ff=only vs. pull.rebase) * --rebase[=!false] vs. --no-ff and --ff (and the related pull.rebase=!false overrides pull.ff=!only) * command line flags take precedence over config, except: * --no-rebase heeds pull.ff=!only * pull.rebase=!false vs --no-ff and --ff For more details behind these rules and a larger table of individual cases, refer to https://lore.kernel.org/git/xmqqwnpqot4m.fsf@gitster.g/ and the links found therein. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- t/t7601-merge-pull-config.sh | 182 +++++++++++++++++++++++++++++++++++ 1 file changed, 182 insertions(+) diff --git a/t/t7601-merge-pull-config.sh b/t/t7601-merge-pull-config.sh index 52e8ccc933..a44e6e69f2 100755 --- a/t/t7601-merge-pull-config.sh +++ b/t/t7601-merge-pull-config.sh @@ -143,6 +143,188 @@ test_expect_success 'pull.rebase not set and --ff-only given (not-fast-forward)' test_i18ngrep ! "Pulling without specifying how to reconcile" err ' +test_does_rebase () { + git reset --hard c2 && + git "$@" . c1 && + # Check that we actually did a rebase + git rev-list --count HEAD >actual && + git rev-list --merges --count HEAD >>actual && + test_write_lines 3 0 >expect && + test_cmp expect actual && + rm actual expect +} + +# Prefers merge over fast-forward +test_does_merge_when_ff_possible () { + git reset --hard c0 && + git "$@" . c1 && + # Check that we actually did a merge + git rev-list --count HEAD >actual && + git rev-list --merges --count HEAD >>actual && + test_write_lines 3 1 >expect && + test_cmp expect actual && + rm actual expect +} + +# Prefers fast-forward over merge or rebase +test_does_fast_forward () { + git reset --hard c0 && + git "$@" . c1 && + + # Check that we did not get any merges + git rev-list --count HEAD >actual && + git rev-list --merges --count HEAD >>actual && + test_write_lines 2 0 >expect && + test_cmp expect actual && + + # Check that we ended up at c1 + git rev-parse HEAD >actual && + git rev-parse c1^{commit} >expect && + test_cmp actual expect && + + # Remove temporary files + rm actual expect +} + +# Doesn't fail when fast-forward not possible; does a merge +test_falls_back_to_full_merge () { + git reset --hard c2 && + git "$@" . c1 && + # Check that we actually did a merge + git rev-list --count HEAD >actual && + git rev-list --merges --count HEAD >>actual && + test_write_lines 4 1 >expect && + test_cmp expect actual && + rm actual expect +} + +# Attempts fast forward, which is impossible, and bails +test_attempts_fast_forward () { + git reset --hard c2 && + test_must_fail git "$@" . c1 2>err && + test_i18ngrep "Not possible to fast-forward, aborting" err +} + +# +# Group 1: Interaction of --ff-only with --[no-]rebase +# (And related interaction of pull.ff=only with pull.rebase) +# +test_expect_failure '--ff-only overrides --rebase' ' + test_attempts_fast_forward pull --rebase --ff-only +' + +test_expect_failure '--ff-only overrides --rebase even if first' ' + test_attempts_fast_forward pull --ff-only --rebase +' + +test_expect_success '--ff-only overrides --no-rebase' ' + test_attempts_fast_forward pull --ff-only --no-rebase +' + +test_expect_failure 'pull.ff=only overrides pull.rebase=true' ' + test_attempts_fast_forward -c pull.ff=only -c pull.rebase=true pull +' + +test_expect_success 'pull.ff=only overrides pull.rebase=false' ' + test_attempts_fast_forward -c pull.ff=only -c pull.rebase=false pull +' + +# Group 2: --rebase=[!false] overrides --no-ff and --ff +# (And related interaction of pull.rebase=!false and pull.ff=!only) +test_expect_success '--rebase overrides --no-ff' ' + test_does_rebase pull --rebase --no-ff +' + +test_expect_success '--rebase overrides --ff' ' + test_does_rebase pull --rebase --ff +' + +test_expect_success '--rebase fast-forwards when possible' ' + test_does_fast_forward pull --rebase --ff +' + +test_expect_success 'pull.rebase=true overrides pull.ff=false' ' + test_does_rebase -c pull.rebase=true -c pull.ff=false pull +' + +test_expect_success 'pull.rebase=true overrides pull.ff=true' ' + test_does_rebase -c pull.rebase=true -c pull.ff=true pull +' + +# Group 3: command line flags take precedence over config +test_expect_failure '--ff-only takes precedence over pull.rebase=true' ' + test_attempts_fast_forward -c pull.rebase=true pull --ff-only +' + +test_expect_success '--ff-only takes precedence over pull.rebase=false' ' + test_attempts_fast_forward -c pull.rebase=false pull --ff-only +' + +test_expect_failure '--no-rebase takes precedence over pull.ff=only' ' + test_falls_back_to_full_merge -c pull.ff=only pull --no-rebase +' + +test_expect_success '--rebase takes precedence over pull.ff=only' ' + test_does_rebase -c pull.ff=only pull --rebase +' + +test_expect_success '--rebase overrides pull.ff=true' ' + test_does_rebase -c pull.ff=true pull --rebase +' + +test_expect_success '--rebase overrides pull.ff=false' ' + test_does_rebase -c pull.ff=false pull --rebase +' + +test_expect_success '--rebase overrides pull.ff unset' ' + test_does_rebase pull --rebase +' + +# Group 4: --no-rebase heeds pull.ff=!only or explict --ff or --no-ff + +test_expect_success '--no-rebase works with --no-ff' ' + test_does_merge_when_ff_possible pull --no-rebase --no-ff +' + +test_expect_success '--no-rebase works with --ff' ' + test_does_fast_forward pull --no-rebase --ff +' + +test_expect_success '--no-rebase does ff if pull.ff unset' ' + test_does_fast_forward pull --no-rebase +' + +test_expect_success '--no-rebase heeds pull.ff=true' ' + test_does_fast_forward -c pull.ff=true pull --no-rebase +' + +test_expect_success '--no-rebase heeds pull.ff=false' ' + test_does_merge_when_ff_possible -c pull.ff=false pull --no-rebase +' + +# Group 5: pull.rebase=!false in combination with --no-ff or --ff +test_expect_success 'pull.rebase=true and --no-ff' ' + test_does_rebase -c pull.rebase=true pull --no-ff +' + +test_expect_success 'pull.rebase=true and --ff' ' + test_does_rebase -c pull.rebase=true pull --ff +' + +test_expect_success 'pull.rebase=false and --no-ff' ' + test_does_merge_when_ff_possible -c pull.rebase=false pull --no-ff +' + +test_expect_success 'pull.rebase=false and --ff, ff possible' ' + test_does_fast_forward -c pull.rebase=false pull --ff +' + +test_expect_success 'pull.rebase=false and --ff, ff not possible' ' + test_falls_back_to_full_merge -c pull.rebase=false pull --ff +' + +# End of groupings for conflicting merge vs. rebase flags/options + test_expect_success 'merge c1 with c2' ' git reset --hard c1 && test -f c0.c && From 1d25e5bdf5263a227d233eafcf028b36c72716f6 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Wed, 21 Jul 2021 01:42:18 +0000 Subject: [PATCH 010/762] t7601: add tests of interactions with multiple merge heads and config There were already code checking that --rebase was incompatible with a merge of multiple heads. However, we were sometimes throwing warnings about lack of specification of rebase vs. merge when given multiple heads. Since rebasing is disallowed with multiple merge heads, that seems like a poor warning to print; we should instead just assume merging is wanted. Add a few tests checking multiple merge head behavior. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- t/t7601-merge-pull-config.sh | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/t/t7601-merge-pull-config.sh b/t/t7601-merge-pull-config.sh index a44e6e69f2..21db1e9e14 100755 --- a/t/t7601-merge-pull-config.sh +++ b/t/t7601-merge-pull-config.sh @@ -325,6 +325,28 @@ test_expect_success 'pull.rebase=false and --ff, ff not possible' ' # End of groupings for conflicting merge vs. rebase flags/options +test_expect_failure 'Multiple heads warns about inability to fast forward' ' + git reset --hard c1 && + test_must_fail git pull . c2 c3 2>err && + test_i18ngrep "Pulling without specifying how to reconcile" err +' + +test_expect_failure 'Multiple can never be fast forwarded' ' + git reset --hard c0 && + test_must_fail git -c pull.ff=only pull . c1 c2 c3 2>err && + test_i18ngrep ! "Pulling without specifying how to reconcile" err && + # In addition to calling out "cannot fast-forward", we very much + # want the "multiple branches" piece to be called out to users. + test_i18ngrep "Cannot fast-forward to multiple branches" err +' + +test_expect_success 'Cannot rebase with multiple heads' ' + git reset --hard c0 && + test_must_fail git -c pull.rebase=true pull . c1 c2 c3 2>err && + test_i18ngrep ! "Pulling without specifying how to reconcile" err && + test_i18ngrep "Cannot rebase onto multiple branches." err +' + test_expect_success 'merge c1 with c2' ' git reset --hard c1 && test -f c0.c && From 3d5fc24daefbdf56bc36a491aed0b7990fa0c62f Mon Sep 17 00:00:00 2001 From: Alex Henrie Date: Wed, 21 Jul 2021 01:42:19 +0000 Subject: [PATCH 011/762] pull: abort if --ff-only is given and fast-forwarding is impossible The warning about pulling without specifying how to reconcile divergent branches says that after setting pull.rebase to true, --ff-only can still be passed on the command line to require a fast-forward. Make that actually work. Signed-off-by: Alex Henrie [en: updated tests; note 3 fixes and 1 new failure] Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- advice.c | 5 +++++ advice.h | 1 + builtin/merge.c | 2 +- builtin/pull.c | 11 ++++++++--- t/t7601-merge-pull-config.sh | 10 +++++----- 5 files changed, 20 insertions(+), 9 deletions(-) diff --git a/advice.c b/advice.c index 0b9c89c48a..337e8f342b 100644 --- a/advice.c +++ b/advice.c @@ -286,6 +286,11 @@ void NORETURN die_conclude_merge(void) die(_("Exiting because of unfinished merge.")); } +void NORETURN die_ff_impossible(void) +{ + die(_("Not possible to fast-forward, aborting.")); +} + void advise_on_updating_sparse_paths(struct string_list *pathspec_list) { struct string_list_item *item; diff --git a/advice.h b/advice.h index bd26c385d0..1624043838 100644 --- a/advice.h +++ b/advice.h @@ -95,6 +95,7 @@ void advise_if_enabled(enum advice_type type, const char *advice, ...); int error_resolve_conflict(const char *me); void NORETURN die_resolve_conflict(const char *me); void NORETURN die_conclude_merge(void); +void NORETURN die_ff_impossible(void); void advise_on_updating_sparse_paths(struct string_list *pathspec_list); void detach_advice(const char *new_name); diff --git a/builtin/merge.c b/builtin/merge.c index a8a843b1f5..aa920ac524 100644 --- a/builtin/merge.c +++ b/builtin/merge.c @@ -1620,7 +1620,7 @@ int cmd_merge(int argc, const char **argv, const char *prefix) } if (fast_forward == FF_ONLY) - die(_("Not possible to fast-forward, aborting.")); + die_ff_impossible(); if (autostash) create_autostash(the_repository, diff --git a/builtin/pull.c b/builtin/pull.c index 3e13f81084..d979660482 100644 --- a/builtin/pull.c +++ b/builtin/pull.c @@ -1046,9 +1046,14 @@ int cmd_pull(int argc, const char **argv, const char *prefix) can_ff = get_can_ff(&orig_head, &merge_heads.oid[0]); - if (rebase_unspecified && !opt_ff && !can_ff) { - if (opt_verbosity >= 0) - show_advice_pull_non_ff(); + if (!can_ff) { + if (opt_ff) { + if (!strcmp(opt_ff, "--ff-only")) + die_ff_impossible(); + } else { + if (rebase_unspecified && opt_verbosity >= 0) + show_advice_pull_non_ff(); + } } if (opt_rebase) { diff --git a/t/t7601-merge-pull-config.sh b/t/t7601-merge-pull-config.sh index 21db1e9e14..d1f621725a 100755 --- a/t/t7601-merge-pull-config.sh +++ b/t/t7601-merge-pull-config.sh @@ -209,11 +209,11 @@ test_attempts_fast_forward () { # Group 1: Interaction of --ff-only with --[no-]rebase # (And related interaction of pull.ff=only with pull.rebase) # -test_expect_failure '--ff-only overrides --rebase' ' +test_expect_success '--ff-only overrides --rebase' ' test_attempts_fast_forward pull --rebase --ff-only ' -test_expect_failure '--ff-only overrides --rebase even if first' ' +test_expect_success '--ff-only overrides --rebase even if first' ' test_attempts_fast_forward pull --ff-only --rebase ' @@ -221,7 +221,7 @@ test_expect_success '--ff-only overrides --no-rebase' ' test_attempts_fast_forward pull --ff-only --no-rebase ' -test_expect_failure 'pull.ff=only overrides pull.rebase=true' ' +test_expect_success 'pull.ff=only overrides pull.rebase=true' ' test_attempts_fast_forward -c pull.ff=only -c pull.rebase=true pull ' @@ -252,7 +252,7 @@ test_expect_success 'pull.rebase=true overrides pull.ff=true' ' ' # Group 3: command line flags take precedence over config -test_expect_failure '--ff-only takes precedence over pull.rebase=true' ' +test_expect_success '--ff-only takes precedence over pull.rebase=true' ' test_attempts_fast_forward -c pull.rebase=true pull --ff-only ' @@ -264,7 +264,7 @@ test_expect_failure '--no-rebase takes precedence over pull.ff=only' ' test_falls_back_to_full_merge -c pull.ff=only pull --no-rebase ' -test_expect_success '--rebase takes precedence over pull.ff=only' ' +test_expect_failure '--rebase takes precedence over pull.ff=only' ' test_does_rebase -c pull.ff=only pull --rebase ' From 866a3014de34d415300bdd488f2ccd7ae5fb2d65 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Thu, 22 Jul 2021 00:57:40 +0200 Subject: [PATCH 012/762] test-lib tests: move "run_sub_test" to a new lib-subtest.sh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Move the "check_sub_test_lib_test()" and its sister functions to a new lib-subtest.sh. In the future (not in this series) I'd like to test test-lib's output in a more targeted and smaller test, and I'll need these functions to do that. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- t/lib-subtest.sh | 86 ++++++++++++++++++++++++++++++++++++++++++++++ t/t0000-basic.sh | 88 +----------------------------------------------- 2 files changed, 87 insertions(+), 87 deletions(-) create mode 100644 t/lib-subtest.sh diff --git a/t/lib-subtest.sh b/t/lib-subtest.sh new file mode 100644 index 0000000000..3cfe09911a --- /dev/null +++ b/t/lib-subtest.sh @@ -0,0 +1,86 @@ +_run_sub_test_lib_test_common () { + neg="$1" name="$2" descr="$3" # stdin is the body of the test code + shift 3 + + # intercept pseudo-options at the front of the argument list that we + # will not pass to child script + skip= + while test $# -gt 0 + do + case "$1" in + --skip=*) + skip=${1#--*=} + shift + ;; + *) + break + ;; + esac + done + + mkdir "$name" && + ( + # Pretend we're not running under a test harness, whether we + # are or not. The test-lib output depends on the setting of + # this variable, so we need a stable setting under which to run + # the sub-test. + sane_unset HARNESS_ACTIVE && + cd "$name" && + write_script "$name.sh" "$TEST_SHELL_PATH" <<-EOF && + test_description='$descr (run in sub test-lib) + + This is run in a sub test-lib so that we do not get incorrect + passing metrics + ' + + # Point to the t/test-lib.sh, which isn't in ../ as usual + . "\$TEST_DIRECTORY"/test-lib.sh + EOF + cat >>"$name.sh" && + export TEST_DIRECTORY && + # The child test re-sources GIT-BUILD-OPTIONS and may thus + # override the test output directory. We thus pass it as an + # explicit override to the child. + TEST_OUTPUT_DIRECTORY_OVERRIDE=$(pwd) && + export TEST_OUTPUT_DIRECTORY_OVERRIDE && + GIT_SKIP_TESTS=$skip && + export GIT_SKIP_TESTS && + sane_unset GIT_TEST_FAIL_PREREQS && + if test -z "$neg" + then + ./"$name.sh" "$@" >out 2>err + else + ! ./"$name.sh" "$@" >out 2>err + fi + ) +} + +run_sub_test_lib_test () { + _run_sub_test_lib_test_common '' "$@" +} + +run_sub_test_lib_test_err () { + _run_sub_test_lib_test_common '!' "$@" +} + +check_sub_test_lib_test () { + name="$1" # stdin is the expected output from the test + ( + cd "$name" && + test_must_be_empty err && + sed -e 's/^> //' -e 's/Z$//' >expect && + test_cmp expect out + ) +} + +check_sub_test_lib_test_err () { + name="$1" # stdin is the expected output from the test + # expected error output is in descriptor 3 + ( + cd "$name" && + sed -e 's/^> //' -e 's/Z$//' >expect.out && + test_cmp expect.out out && + sed -e 's/^> //' -e 's/Z$//' <&3 >expect.err && + test_cmp expect.err err + ) +} diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh index cb87768513..a3865dd77b 100755 --- a/t/t0000-basic.sh +++ b/t/t0000-basic.sh @@ -19,6 +19,7 @@ modification *should* take notice and update the test vectors here. ' . ./test-lib.sh +. "$TEST_DIRECTORY"/lib-subtest.sh try_local_xy () { local x="local" y="alsolocal" && @@ -66,93 +67,6 @@ test_expect_success 'success is reported like this' ' : ' -_run_sub_test_lib_test_common () { - neg="$1" name="$2" descr="$3" # stdin is the body of the test code - shift 3 - - # intercept pseudo-options at the front of the argument list that we - # will not pass to child script - skip= - while test $# -gt 0 - do - case "$1" in - --skip=*) - skip=${1#--*=} - shift - ;; - *) - break - ;; - esac - done - - mkdir "$name" && - ( - # Pretend we're not running under a test harness, whether we - # are or not. The test-lib output depends on the setting of - # this variable, so we need a stable setting under which to run - # the sub-test. - sane_unset HARNESS_ACTIVE && - cd "$name" && - write_script "$name.sh" "$TEST_SHELL_PATH" <<-EOF && - test_description='$descr (run in sub test-lib) - - This is run in a sub test-lib so that we do not get incorrect - passing metrics - ' - - # Point to the t/test-lib.sh, which isn't in ../ as usual - . "\$TEST_DIRECTORY"/test-lib.sh - EOF - cat >>"$name.sh" && - export TEST_DIRECTORY && - # The child test re-sources GIT-BUILD-OPTIONS and may thus - # override the test output directory. We thus pass it as an - # explicit override to the child. - TEST_OUTPUT_DIRECTORY_OVERRIDE=$(pwd) && - export TEST_OUTPUT_DIRECTORY_OVERRIDE && - GIT_SKIP_TESTS=$skip && - export GIT_SKIP_TESTS && - sane_unset GIT_TEST_FAIL_PREREQS && - if test -z "$neg" - then - ./"$name.sh" "$@" >out 2>err - else - ! ./"$name.sh" "$@" >out 2>err - fi - ) -} - -run_sub_test_lib_test () { - _run_sub_test_lib_test_common '' "$@" -} - -run_sub_test_lib_test_err () { - _run_sub_test_lib_test_common '!' "$@" -} - -check_sub_test_lib_test () { - name="$1" # stdin is the expected output from the test - ( - cd "$name" && - test_must_be_empty err && - sed -e 's/^> //' -e 's/Z$//' >expect && - test_cmp expect out - ) -} - -check_sub_test_lib_test_err () { - name="$1" # stdin is the expected output from the test - # expected error output is in descriptor 3 - ( - cd "$name" && - sed -e 's/^> //' -e 's/Z$//' >expect.out && - test_cmp expect.out out && - sed -e 's/^> //' -e 's/Z$//' <&3 >expect.err && - test_cmp expect.err err - ) -} - test_expect_success 'pretend we have a fully passing test suite' ' run_sub_test_lib_test full-pass "3 passing tests" <<-\EOF && for i in 1 2 3 From 7171221d82249cb8ce4b73a40852a487d1cebde7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Thu, 22 Jul 2021 01:23:04 +0200 Subject: [PATCH 013/762] Makefile: don't use "FORCE" for tags targets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove the "FORCE" dependency from the "tags", "TAGS" and "cscope" targets, instead make them depend on whether or not the relevant source files have changed. For the cscope target we need to change it to depend on the actual generated file while we generate while we're at it, as the next commit will discuss we always generate a cscope.out file. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Makefile | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/Makefile b/Makefile index faa8900097..55d0a98dc6 100644 --- a/Makefile +++ b/Makefile @@ -2727,20 +2727,24 @@ FIND_SOURCE_FILES = ( \ | sed -e 's|^\./||' \ ) -$(ETAGS_TARGET): FORCE +FOUND_SOURCE_FILES = $(shell $(FIND_SOURCE_FILES)) + +$(ETAGS_TARGET): $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) "$(ETAGS_TARGET)+" && \ - $(FIND_SOURCE_FILES) | xargs etags -a -o "$(ETAGS_TARGET)+" && \ + echo $(FOUND_SOURCE_FILES) | xargs etags -a -o "$(ETAGS_TARGET)+" && \ mv "$(ETAGS_TARGET)+" "$(ETAGS_TARGET)" -tags: FORCE +tags: $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) tags+ && \ - $(FIND_SOURCE_FILES) | xargs ctags -a -o tags+ && \ + echo $(FOUND_SOURCE_FILES) | xargs ctags -a -o tags+ && \ mv tags+ tags -.PHONY: cscope -cscope: +cscope.out: $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) cscope* && \ - $(FIND_SOURCE_FILES) | xargs cscope -b + echo $(FOUND_SOURCE_FILES) | xargs cscope -b + +.PHONY: cscope +cscope: cscope.out ### Detect prefix changes TRACK_PREFIX = $(bindir_SQ):$(gitexecdir_SQ):$(template_dir_SQ):$(prefix_SQ):\ @@ -2921,7 +2925,7 @@ check: config-list.h command-list.h exit 1; \ fi -FOUND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) +FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES)) COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) %.cocci.patch: %.cocci $(COCCI_SOURCES) From e4dc25ed49d94c93b1d3f63efe17f32ca682cab7 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Thu, 22 Jul 2021 05:04:46 +0000 Subject: [PATCH 014/762] pull: since --ff-only overrides, handle it first There are both merge and rebase branches in the logic, and previously both had to handle fast-forwarding. Merge handled that implicitly (because git merge handles it directly), while in rebase it was explicit. Given that the --ff-only flag is meant to override any --rebase or --no-rebase, make the code reflect that by handling --ff-only before the merge-vs-rebase logic. It turns out that this also fixes a bug for submodules. Previously, when --ff-only was given, the code would run `merge --ff-only` on the main module, and then run `submodule update --recursive --rebase` on the submodules. With this change, we still run `merge --ff-only` on the main module, but now run `submodule update --recursive --checkout` on the submodules. I believe this better reflects the intent of --ff-only to have it apply to both the main module and the submodules. (Sidenote: It is somewhat interesting that all merges pass `--checkout` to submodule update, even when `--no-ff` is specified, meaning that it will only do fast-forward merges for submodules. This was discussed in commit a6d7eb2c7a ("pull: optionally rebase submodules (remote submodule changes only)", 2017-06-23). The same limitations apply now as then, so we are not trying to fix this at this time.) Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- builtin/pull.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/builtin/pull.c b/builtin/pull.c index d979660482..1f45202037 100644 --- a/builtin/pull.c +++ b/builtin/pull.c @@ -1046,15 +1046,15 @@ int cmd_pull(int argc, const char **argv, const char *prefix) can_ff = get_can_ff(&orig_head, &merge_heads.oid[0]); - if (!can_ff) { - if (opt_ff) { - if (!strcmp(opt_ff, "--ff-only")) - die_ff_impossible(); - } else { - if (rebase_unspecified && opt_verbosity >= 0) - show_advice_pull_non_ff(); - } + /* ff-only takes precedence over rebase */ + if (opt_ff && !strcmp(opt_ff, "--ff-only")) { + if (!can_ff) + die_ff_impossible(); + opt_rebase = REBASE_FALSE; } + /* If no action specified and we can't fast forward, then warn. */ + if (!opt_ff && rebase_unspecified && !can_ff) + show_advice_pull_non_ff(); if (opt_rebase) { int ret = 0; From adc27d6a9374d012b091bc348c20f5bfdbee52d1 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Thu, 22 Jul 2021 05:04:47 +0000 Subject: [PATCH 015/762] pull: make --rebase and --no-rebase override pull.ff=only Fix the last few precedence tests failing in t7601 by now implementing the logic to have --[no-]rebase override a pull.ff=only config setting. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- builtin/pull.c | 16 +++++++++++++++- t/t7601-merge-pull-config.sh | 4 ++-- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/builtin/pull.c b/builtin/pull.c index 1f45202037..9bf0325529 100644 --- a/builtin/pull.c +++ b/builtin/pull.c @@ -966,8 +966,22 @@ int cmd_pull(int argc, const char **argv, const char *prefix) parse_repo_refspecs(argc, argv, &repo, &refspecs); - if (!opt_ff) + if (!opt_ff) { opt_ff = xstrdup_or_null(config_get_ff()); + /* + * A subtle point: opt_ff was set on the line above via + * reading from config. opt_rebase, in contrast, is set + * before this point via command line options. The setting + * of opt_rebase via reading from config (using + * config_get_rebase()) does not happen until later. We + * are relying on the next if-condition happening before + * the config_get_rebase() call so that an explicit + * "--rebase" can override a config setting of + * pull.ff=only. + */ + if (opt_rebase >= 0 && opt_ff && !strcmp(opt_ff, "--ff-only")) + opt_ff = "--ff"; + } if (opt_rebase < 0) opt_rebase = config_get_rebase(&rebase_unspecified); diff --git a/t/t7601-merge-pull-config.sh b/t/t7601-merge-pull-config.sh index d1f621725a..0c0dfecd2a 100755 --- a/t/t7601-merge-pull-config.sh +++ b/t/t7601-merge-pull-config.sh @@ -260,11 +260,11 @@ test_expect_success '--ff-only takes precedence over pull.rebase=false' ' test_attempts_fast_forward -c pull.rebase=false pull --ff-only ' -test_expect_failure '--no-rebase takes precedence over pull.ff=only' ' +test_expect_success '--no-rebase takes precedence over pull.ff=only' ' test_falls_back_to_full_merge -c pull.ff=only pull --no-rebase ' -test_expect_failure '--rebase takes precedence over pull.ff=only' ' +test_expect_success '--rebase takes precedence over pull.ff=only' ' test_does_rebase -c pull.ff=only pull --rebase ' From 031e2f7ae195069d00d21cde906fce5b0318dbdd Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Thu, 22 Jul 2021 05:04:48 +0000 Subject: [PATCH 016/762] pull: abort by default when fast-forwarding is not possible We have for some time shown a long warning when the user does not specify how to reconcile divergent branches with git pull. Make it an error now. Initial-patch-by: Alex Henrie Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- Documentation/git-pull.txt | 18 ++++++++----- builtin/pull.c | 10 +++++--- t/t4013-diff-various.sh | 2 +- t/t5520-pull.sh | 20 +++++++-------- t/t5521-pull-options.sh | 4 +-- t/t5524-pull-msg.sh | 4 +-- t/t5553-set-upstream.sh | 14 +++++----- t/t5604-clone-reference.sh | 4 +-- t/t6402-merge-rename.sh | 18 ++++++------- t/t6409-merge-subtree.sh | 6 ++--- t/t6417-merge-ours-theirs.sh | 10 ++++---- t/t7601-merge-pull-config.sh | 48 +++++++++++++++++------------------ t/t7603-merge-reduce-heads.sh | 2 +- 13 files changed, 83 insertions(+), 77 deletions(-) diff --git a/Documentation/git-pull.txt b/Documentation/git-pull.txt index 5c3fb67c01..cad3f6bfce 100644 --- a/Documentation/git-pull.txt +++ b/Documentation/git-pull.txt @@ -15,14 +15,18 @@ SYNOPSIS DESCRIPTION ----------- -Incorporates changes from a remote repository into the current -branch. In its default mode, `git pull` is shorthand for -`git fetch` followed by `git merge FETCH_HEAD`. +Incorporates changes from a remote repository into the current branch. +If the current branch is behind the remote, then by default it will +fast-forward the current branch to match the remote. If the current +branch and the remote have diverged, the user needs to specify how to +reconcile the divergent branches with `--no-ff`, `--ff`, or `--rebase` +(or the corresponding configuration options in `pull.ff` or +`pull.rebase`). -More precisely, 'git pull' runs 'git fetch' with the given -parameters and calls 'git merge' to merge the retrieved branch -heads into the current branch. -With `--rebase`, it runs 'git rebase' instead of 'git merge'. +More precisely, `git pull` runs `git fetch` with the given parameters +and then depending on configuration options or command line flags, +will call either `git merge` or `git rebase` to reconcile diverging +branches. should be the name of a remote repository as passed to linkgit:git-fetch[1]. can name an diff --git a/builtin/pull.c b/builtin/pull.c index 9bf0325529..4514a1478e 100644 --- a/builtin/pull.c +++ b/builtin/pull.c @@ -927,9 +927,9 @@ static int get_can_ff(struct object_id *orig_head, struct object_id *orig_merge_ static void show_advice_pull_non_ff(void) { - advise(_("Pulling without specifying how to reconcile divergent branches is\n" - "discouraged. You can squelch this message by running one of the following\n" - "commands sometime before your next pull:\n" + advise(_("You have divergent branches and need to specify how to reconcile them.\n" + "You can do so by running one of the following commands sometime before\n" + "your next pull:\n" "\n" " git config pull.rebase false # merge (the default strategy)\n" " git config pull.rebase true # rebase\n" @@ -1067,8 +1067,10 @@ int cmd_pull(int argc, const char **argv, const char *prefix) opt_rebase = REBASE_FALSE; } /* If no action specified and we can't fast forward, then warn. */ - if (!opt_ff && rebase_unspecified && !can_ff) + if (!opt_ff && rebase_unspecified && !can_ff) { show_advice_pull_non_ff(); + die(_("Need to specify how to reconcile divergent branches.")); + } if (opt_rebase) { int ret = 0; diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh index 7fadc985cc..eb989f7f19 100755 --- a/t/t4013-diff-various.sh +++ b/t/t4013-diff-various.sh @@ -65,7 +65,7 @@ test_expect_success setup ' export GIT_AUTHOR_DATE GIT_COMMITTER_DATE && git checkout master && - git pull -s ours . side && + git pull -s ours --no-rebase . side && GIT_AUTHOR_DATE="2006-06-26 00:05:00 +0000" && GIT_COMMITTER_DATE="2006-06-26 00:05:00 +0000" && diff --git a/t/t5520-pull.sh b/t/t5520-pull.sh index e2c0c51022..672001a18b 100755 --- a/t/t5520-pull.sh +++ b/t/t5520-pull.sh @@ -136,12 +136,12 @@ test_expect_success 'the default remote . should not break explicit pull' ' git reset --hard HEAD^ && echo file >expect && test_cmp expect file && - git pull . second && + git pull --no-rebase . second && echo modified >expect && test_cmp expect file && git reflog -1 >reflog.actual && sed "s/^[0-9a-f][0-9a-f]*/OBJID/" reflog.actual >reflog.fuzzy && - echo "OBJID HEAD@{0}: pull . second: Fast-forward" >reflog.expected && + echo "OBJID HEAD@{0}: pull --no-rebase . second: Fast-forward" >reflog.expected && test_cmp reflog.expected reflog.fuzzy ' @@ -226,7 +226,7 @@ test_expect_success 'fail if the index has unresolved entries' ' test_commit modified2 file && git ls-files -u >unmerged && test_must_be_empty unmerged && - test_must_fail git pull . second && + test_must_fail git pull --no-rebase . second && git ls-files -u >unmerged && test_file_not_empty unmerged && cp file expected && @@ -409,37 +409,37 @@ test_expect_success 'pull --rebase --no-autostash & rebase.autostash unset' ' test_expect_success 'pull succeeds with dirty working directory and merge.autostash set' ' test_config merge.autostash true && - test_pull_autostash 2 + test_pull_autostash 2 --no-rebase ' test_expect_success 'pull --autostash & merge.autostash=true' ' test_config merge.autostash true && - test_pull_autostash 2 --autostash + test_pull_autostash 2 --autostash --no-rebase ' test_expect_success 'pull --autostash & merge.autostash=false' ' test_config merge.autostash false && - test_pull_autostash 2 --autostash + test_pull_autostash 2 --autostash --no-rebase ' test_expect_success 'pull --autostash & merge.autostash unset' ' test_unconfig merge.autostash && - test_pull_autostash 2 --autostash + test_pull_autostash 2 --autostash --no-rebase ' test_expect_success 'pull --no-autostash & merge.autostash=true' ' test_config merge.autostash true && - test_pull_autostash_fail --no-autostash + test_pull_autostash_fail --no-autostash --no-rebase ' test_expect_success 'pull --no-autostash & merge.autostash=false' ' test_config merge.autostash false && - test_pull_autostash_fail --no-autostash + test_pull_autostash_fail --no-autostash --no-rebase ' test_expect_success 'pull --no-autostash & merge.autostash unset' ' test_unconfig merge.autostash && - test_pull_autostash_fail --no-autostash + test_pull_autostash_fail --no-autostash --no-rebase ' test_expect_success 'pull.rebase' ' diff --git a/t/t5521-pull-options.sh b/t/t5521-pull-options.sh index 63a688bdbf..7601c919fd 100755 --- a/t/t5521-pull-options.sh +++ b/t/t5521-pull-options.sh @@ -113,7 +113,7 @@ test_expect_success 'git pull --force' ' git pull two && test_commit A && git branch -f origin && - git pull --all --force + git pull --no-rebase --all --force ) ' @@ -179,7 +179,7 @@ test_expect_success 'git pull --allow-unrelated-histories' ' ( cd dst && test_must_fail git pull ../src side && - git pull --allow-unrelated-histories ../src side + git pull --no-rebase --allow-unrelated-histories ../src side ) ' diff --git a/t/t5524-pull-msg.sh b/t/t5524-pull-msg.sh index c278adaa5a..b2be3605f5 100755 --- a/t/t5524-pull-msg.sh +++ b/t/t5524-pull-msg.sh @@ -28,7 +28,7 @@ test_expect_success setup ' test_expect_success pull ' ( cd cloned && - git pull --log && + git pull --no-rebase --log && git log -2 && git cat-file commit HEAD >result && grep Dollar result @@ -41,7 +41,7 @@ test_expect_success '--log=1 limits shortlog length' ' git reset --hard HEAD^ && test "$(cat afile)" = original && test "$(cat bfile)" = added && - git pull --log=1 && + git pull --no-rebase --log=1 && git log -3 && git cat-file commit HEAD >result && grep Dollar result && diff --git a/t/t5553-set-upstream.sh b/t/t5553-set-upstream.sh index b1d614ce18..9c12c0f8c3 100755 --- a/t/t5553-set-upstream.sh +++ b/t/t5553-set-upstream.sh @@ -108,27 +108,27 @@ test_expect_success 'setup commit on main and other pull' ' test_expect_success 'pull --set-upstream upstream main sets branch main but not other' ' clear_config main other && - git pull --set-upstream upstream main && + git pull --no-rebase --set-upstream upstream main && check_config main upstream refs/heads/main && check_config_missing other ' test_expect_success 'pull --set-upstream main:other2 does not set the branch other2' ' clear_config other2 && - git pull --set-upstream upstream main:other2 && + git pull --no-rebase --set-upstream upstream main:other2 && check_config_missing other2 ' test_expect_success 'pull --set-upstream upstream other sets branch main' ' clear_config main other && - git pull --set-upstream upstream other && + git pull --no-rebase --set-upstream upstream other && check_config main upstream refs/heads/other && check_config_missing other ' test_expect_success 'pull --set-upstream upstream tag does not set the tag' ' clear_config three && - git pull --tags --set-upstream upstream three && + git pull --no-rebase --tags --set-upstream upstream three && check_config_missing three ' @@ -144,16 +144,16 @@ test_expect_success 'pull --set-upstream http://nosuchdomain.example.com fails w test_expect_success 'pull --set-upstream upstream HEAD sets branch HEAD' ' clear_config main other && - git pull --set-upstream upstream HEAD && + git pull --no-rebase --set-upstream upstream HEAD && check_config main upstream HEAD && git checkout other && - git pull --set-upstream upstream HEAD && + git pull --no-rebase --set-upstream upstream HEAD && check_config other upstream HEAD ' test_expect_success 'pull --set-upstream upstream with more than one branch does nothing' ' clear_config main three && - git pull --set-upstream upstream main three && + git pull --no-rebase --set-upstream upstream main three && check_config_missing main && check_config_missing three ' diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh index e845d621f6..24340e6d56 100755 --- a/t/t5604-clone-reference.sh +++ b/t/t5604-clone-reference.sh @@ -87,7 +87,7 @@ test_expect_success 'updating origin' ' ' test_expect_success 'pulling changes from origin' ' - git -C C pull origin + git -C C pull --no-rebase origin ' # the 2 local objects are commit and tree from the merge @@ -96,7 +96,7 @@ test_expect_success 'that alternate to origin gets used' ' ' test_expect_success 'pulling changes from origin' ' - git -C D pull origin + git -C D pull --no-rebase origin ' # the 5 local objects are expected; file3 blob, commit in A to add it diff --git a/t/t6402-merge-rename.sh b/t/t6402-merge-rename.sh index 425dad97d5..02a842697b 100755 --- a/t/t6402-merge-rename.sh +++ b/t/t6402-merge-rename.sh @@ -103,7 +103,7 @@ test_expect_success 'setup' ' test_expect_success 'pull renaming branch into unrenaming one' \ ' git show-branch && - test_expect_code 1 git pull . white && + test_expect_code 1 git pull --no-rebase . white && git ls-files -s && git ls-files -u B >b.stages && test_line_count = 3 b.stages && @@ -121,7 +121,7 @@ test_expect_success 'pull renaming branch into another renaming one' \ rm -f B && git reset --hard && git checkout red && - test_expect_code 1 git pull . white && + test_expect_code 1 git pull --no-rebase . white && git ls-files -u B >b.stages && test_line_count = 3 b.stages && git ls-files -s N >n.stages && @@ -137,7 +137,7 @@ test_expect_success 'pull unrenaming branch into renaming one' \ ' git reset --hard && git show-branch && - test_expect_code 1 git pull . main && + test_expect_code 1 git pull --no-rebase . main && git ls-files -u B >b.stages && test_line_count = 3 b.stages && git ls-files -s N >n.stages && @@ -153,7 +153,7 @@ test_expect_success 'pull conflicting renames' \ ' git reset --hard && git show-branch && - test_expect_code 1 git pull . blue && + test_expect_code 1 git pull --no-rebase . blue && git ls-files -u A >a.stages && test_line_count = 1 a.stages && git ls-files -u B >b.stages && @@ -173,7 +173,7 @@ test_expect_success 'interference with untracked working tree file' ' git reset --hard && git show-branch && echo >A this file should not matter && - test_expect_code 1 git pull . white && + test_expect_code 1 git pull --no-rebase . white && test_path_is_file A ' @@ -183,7 +183,7 @@ test_expect_success 'interference with untracked working tree file' ' git show-branch && rm -f A && echo >A this file should not matter && - test_expect_code 1 git pull . red && + test_expect_code 1 git pull --no-rebase . red && test_path_is_file A ' @@ -193,7 +193,7 @@ test_expect_success 'interference with untracked working tree file' ' git checkout -f main && git tag -f anchor && git show-branch && - git pull . yellow && + git pull --no-rebase . yellow && test_path_is_missing M && git reset --hard anchor ' @@ -220,7 +220,7 @@ test_expect_success 'updated working tree file should prevent the merge' ' echo >>M one line addition && cat M >M.saved && git update-index M && - test_expect_code 128 git pull . yellow && + test_expect_code 128 git pull --no-rebase . yellow && test_cmp M M.saved && rm -f M.saved ' @@ -232,7 +232,7 @@ test_expect_success 'interference with untracked working tree file' ' git tag -f anchor && git show-branch && echo >M this file should not matter && - git pull . main && + git pull --no-rebase . main && test_path_is_file M && ! { git ls-files -s | diff --git a/t/t6409-merge-subtree.sh b/t/t6409-merge-subtree.sh index d406b2343c..ba7890ec52 100755 --- a/t/t6409-merge-subtree.sh +++ b/t/t6409-merge-subtree.sh @@ -100,7 +100,7 @@ test_expect_success 'merge update' ' git checkout -b topic_2 && git commit -m "update git-gui" && cd ../git && - git pull -s subtree gui topic_2 && + git pull --no-rebase -s subtree gui topic_2 && git ls-files -s >actual && ( echo "100644 $o3 0 git-gui/git-gui.sh" && @@ -129,7 +129,7 @@ test_expect_success 'initial ambiguous subtree' ' test_expect_success 'merge using explicit' ' cd ../git && git reset --hard topic_2 && - git pull -Xsubtree=git-gui gui topic_2 && + git pull --no-rebase -Xsubtree=git-gui gui topic_2 && git ls-files -s >actual && ( echo "100644 $o3 0 git-gui/git-gui.sh" && @@ -142,7 +142,7 @@ test_expect_success 'merge using explicit' ' test_expect_success 'merge2 using explicit' ' cd ../git && git reset --hard topic_2 && - git pull -Xsubtree=git-gui2 gui topic_2 && + git pull --no-rebase -Xsubtree=git-gui2 gui topic_2 && git ls-files -s >actual && ( echo "100644 $o1 0 git-gui/git-gui.sh" && diff --git a/t/t6417-merge-ours-theirs.sh b/t/t6417-merge-ours-theirs.sh index ac9aee9a66..ec065d6a65 100755 --- a/t/t6417-merge-ours-theirs.sh +++ b/t/t6417-merge-ours-theirs.sh @@ -69,11 +69,11 @@ test_expect_success 'binary file with -Xours/-Xtheirs' ' ' test_expect_success 'pull passes -X to underlying merge' ' - git reset --hard main && git pull -s recursive -Xours . side && - git reset --hard main && git pull -s recursive -X ours . side && - git reset --hard main && git pull -s recursive -Xtheirs . side && - git reset --hard main && git pull -s recursive -X theirs . side && - git reset --hard main && test_must_fail git pull -s recursive -X bork . side + git reset --hard main && git pull --no-rebase -s recursive -Xours . side && + git reset --hard main && git pull --no-rebase -s recursive -X ours . side && + git reset --hard main && git pull --no-rebase -s recursive -Xtheirs . side && + git reset --hard main && git pull --no-rebase -s recursive -X theirs . side && + git reset --hard main && test_must_fail git pull --no-rebase -s recursive -X bork . side ' test_expect_success SYMLINKS 'symlink with -Xours/-Xtheirs' ' diff --git a/t/t7601-merge-pull-config.sh b/t/t7601-merge-pull-config.sh index 0c0dfecd2a..742ed3981c 100755 --- a/t/t7601-merge-pull-config.sh +++ b/t/t7601-merge-pull-config.sh @@ -27,120 +27,120 @@ test_expect_success 'setup' ' git tag c3 ' -test_expect_success 'pull.rebase not set' ' +test_expect_success 'pull.rebase not set, ff possible' ' git reset --hard c0 && git pull . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and pull.ff=true' ' git reset --hard c0 && test_config pull.ff true && git pull . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and pull.ff=false' ' git reset --hard c0 && test_config pull.ff false && git pull . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and pull.ff=only' ' git reset --hard c0 && test_config pull.ff only && git pull . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and --rebase given' ' git reset --hard c0 && git pull --rebase . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and --no-rebase given' ' git reset --hard c0 && git pull --no-rebase . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and --ff given' ' git reset --hard c0 && git pull --ff . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and --no-ff given' ' git reset --hard c0 && git pull --no-ff . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and --ff-only given' ' git reset --hard c0 && git pull --ff-only . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set (not-fast-forward)' ' git reset --hard c2 && - git -c color.advice=always pull . c1 2>err && + test_must_fail git -c color.advice=always pull . c1 2>err && test_decode_color decoded && test_i18ngrep "hint: " decoded && - test_i18ngrep "Pulling without specifying how to reconcile" decoded + test_i18ngrep "You have divergent branches" decoded ' test_expect_success 'pull.rebase not set and pull.ff=true (not-fast-forward)' ' git reset --hard c2 && test_config pull.ff true && git pull . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and pull.ff=false (not-fast-forward)' ' git reset --hard c2 && test_config pull.ff false && git pull . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and pull.ff=only (not-fast-forward)' ' git reset --hard c2 && test_config pull.ff only && test_must_fail git pull . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and --rebase given (not-fast-forward)' ' git reset --hard c2 && git pull --rebase . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and --no-rebase given (not-fast-forward)' ' git reset --hard c2 && git pull --no-rebase . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and --ff given (not-fast-forward)' ' git reset --hard c2 && git pull --ff . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and --no-ff given (not-fast-forward)' ' git reset --hard c2 && git pull --no-ff . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_expect_success 'pull.rebase not set and --ff-only given (not-fast-forward)' ' git reset --hard c2 && test_must_fail git pull --ff-only . c1 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err + test_i18ngrep ! "You have divergent branches" err ' test_does_rebase () { @@ -325,16 +325,16 @@ test_expect_success 'pull.rebase=false and --ff, ff not possible' ' # End of groupings for conflicting merge vs. rebase flags/options -test_expect_failure 'Multiple heads warns about inability to fast forward' ' +test_expect_success 'Multiple heads warns about inability to fast forward' ' git reset --hard c1 && test_must_fail git pull . c2 c3 2>err && - test_i18ngrep "Pulling without specifying how to reconcile" err + test_i18ngrep "You have divergent branches" err ' test_expect_failure 'Multiple can never be fast forwarded' ' git reset --hard c0 && test_must_fail git -c pull.ff=only pull . c1 c2 c3 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err && + test_i18ngrep ! "You have divergent branches" err && # In addition to calling out "cannot fast-forward", we very much # want the "multiple branches" piece to be called out to users. test_i18ngrep "Cannot fast-forward to multiple branches" err @@ -343,7 +343,7 @@ test_expect_failure 'Multiple can never be fast forwarded' ' test_expect_success 'Cannot rebase with multiple heads' ' git reset --hard c0 && test_must_fail git -c pull.rebase=true pull . c1 c2 c3 2>err && - test_i18ngrep ! "Pulling without specifying how to reconcile" err && + test_i18ngrep ! "You have divergent branches" err && test_i18ngrep "Cannot rebase onto multiple branches." err ' diff --git a/t/t7603-merge-reduce-heads.sh b/t/t7603-merge-reduce-heads.sh index 98948955ae..27cd94ad6f 100755 --- a/t/t7603-merge-reduce-heads.sh +++ b/t/t7603-merge-reduce-heads.sh @@ -68,7 +68,7 @@ test_expect_success 'merge c1 with c2, c3, c4, c5' ' test_expect_success 'pull c2, c3, c4, c5 into c1' ' git reset --hard c1 && - git pull . c2 c3 c4 c5 && + git pull --no-rebase . c2 c3 c4 c5 && test "$(git rev-parse c1)" != "$(git rev-parse HEAD)" && test "$(git rev-parse c1)" = "$(git rev-parse HEAD^1)" && test "$(git rev-parse c2)" = "$(git rev-parse HEAD^2)" && From 359ff6938990a438b99e95fe36b6b359f3eb9811 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Thu, 22 Jul 2021 05:04:49 +0000 Subject: [PATCH 017/762] pull: update docs & code for option compatibility with rebasing git-pull.txt includes merge-options.txt, which is written assuming merges will happen. git-pull has allowed rebases for many years; update the documentation to reflect that. While at it, pass any `--signoff` flag through to the rebase backend too so that we don't have to document it as merge-specific. Rebase has supported the --signoff flag for years now as well. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- Documentation/git-merge.txt | 2 ++ Documentation/git-pull.txt | 9 ++++---- Documentation/merge-options.txt | 40 +++++++++++++++++++++++++++++++++ builtin/pull.c | 2 ++ 4 files changed, 48 insertions(+), 5 deletions(-) diff --git a/Documentation/git-merge.txt b/Documentation/git-merge.txt index 3819fadac1..e4f3352eb5 100644 --- a/Documentation/git-merge.txt +++ b/Documentation/git-merge.txt @@ -61,6 +61,8 @@ merge has resulted in conflicts. OPTIONS ------- +:git-merge: 1 + include::merge-options.txt[] -m :: diff --git a/Documentation/git-pull.txt b/Documentation/git-pull.txt index cad3f6bfce..077961dfa1 100644 --- a/Documentation/git-pull.txt +++ b/Documentation/git-pull.txt @@ -19,13 +19,12 @@ Incorporates changes from a remote repository into the current branch. If the current branch is behind the remote, then by default it will fast-forward the current branch to match the remote. If the current branch and the remote have diverged, the user needs to specify how to -reconcile the divergent branches with `--no-ff`, `--ff`, or `--rebase` -(or the corresponding configuration options in `pull.ff` or -`pull.rebase`). +reconcile the divergent branches with `--rebase` or `--no-rebase` (or +the corresponding configuration option in `pull.rebase`). More precisely, `git pull` runs `git fetch` with the given parameters and then depending on configuration options or command line flags, -will call either `git merge` or `git rebase` to reconcile diverging +will call either `git rebase` or `git merge` to reconcile diverging branches. should be the name of a remote repository as @@ -136,7 +135,7 @@ published that history already. Do *not* use this option unless you have read linkgit:git-rebase[1] carefully. --no-rebase:: - Override earlier --rebase. + This is shorthand for --rebase=false. Options related to fetching ~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/Documentation/merge-options.txt b/Documentation/merge-options.txt index eb0aabd396..907327bf75 100644 --- a/Documentation/merge-options.txt +++ b/Documentation/merge-options.txt @@ -2,6 +2,9 @@ --no-commit:: Perform the merge and commit the result. This option can be used to override --no-commit. +ifdef::git-pull[] + Only useful when merging. +endif::git-pull[] + With --no-commit perform the merge and stop just before creating a merge commit, to give the user a chance to inspect and further @@ -39,6 +42,7 @@ set to `no` at the beginning of them. to `MERGE_MSG` before being passed on to the commit machinery in the case of a merge conflict. +ifdef::git-merge[] --ff:: --no-ff:: --ff-only:: @@ -47,6 +51,22 @@ set to `no` at the beginning of them. default unless merging an annotated (and possibly signed) tag that is not stored in its natural place in the `refs/tags/` hierarchy, in which case `--no-ff` is assumed. +endif::git-merge[] +ifdef::git-pull[] +--ff-only:: + Only update to the new history if there is no divergent local + history. This is the default when no method for reconciling + divergent histories is provided (via the --rebase=* flags). + +--ff:: +--no-ff:: + When merging rather than rebasing, specifies how a merge is + handled when the merged-in history is already a descendant of + the current history. If merging is requested, `--ff` is the + default unless merging an annotated (and possibly signed) tag + that is not stored in its natural place in the `refs/tags/` + hierarchy, in which case `--no-ff` is assumed. +endif::git-pull[] + With `--ff`, when possible resolve the merge as a fast-forward (only update the branch pointer to match the merged branch; do not create a @@ -55,9 +75,11 @@ descendant of the current history), create a merge commit. + With `--no-ff`, create a merge commit in all cases, even when the merge could instead be resolved as a fast-forward. +ifdef::git-merge[] + With `--ff-only`, resolve the merge as a fast-forward when possible. When not possible, refuse to merge and exit with a non-zero status. +endif::git-merge[] -S[]:: --gpg-sign[=]:: @@ -73,6 +95,9 @@ When not possible, refuse to merge and exit with a non-zero status. In addition to branch names, populate the log message with one-line descriptions from at most actual commits that are being merged. See also linkgit:git-fmt-merge-msg[1]. +ifdef::git-pull[] + Only useful when merging. +endif::git-pull[] + With --no-log do not list one-line descriptions from the actual commits being merged. @@ -102,10 +127,17 @@ With --no-squash perform the merge and commit the result. This option can be used to override --squash. + With --squash, --commit is not allowed, and will fail. +ifdef::git-pull[] ++ +Only useful when merging. +endif::git-pull[] --no-verify:: This option bypasses the pre-merge and commit-msg hooks. See also linkgit:githooks[5]. +ifdef::git-pull[] + Only useful when merging. +endif::git-pull[] -s :: --strategy=:: @@ -127,6 +159,10 @@ With --squash, --commit is not allowed, and will fail. default trust model, this means the signing key has been signed by a trusted key. If the tip commit of the side branch is not signed with a valid key, the merge is aborted. +ifdef::git-pull[] ++ +Only useful when merging. +endif::git-pull[] --summary:: --no-summary:: @@ -166,3 +202,7 @@ endif::git-pull[] projects that started their lives independently. As that is a very rare occasion, no configuration variable to enable this by default exists and will not be added. +ifdef::git-pull[] ++ +Only useful when merging. +endif::git-pull[] diff --git a/builtin/pull.c b/builtin/pull.c index 4514a1478e..2f1d1f4037 100644 --- a/builtin/pull.c +++ b/builtin/pull.c @@ -893,6 +893,8 @@ static int run_rebase(const struct object_id *newbase, strvec_pushv(&args, opt_strategy_opts.v); if (opt_gpg_sign) strvec_push(&args, opt_gpg_sign); + if (opt_signoff) + strvec_push(&args, opt_signoff); if (opt_autostash == 0) strvec_push(&args, "--no-autostash"); else if (opt_autostash == 1) From 6f843a3355ee590dfe09eb90679051e75fadf675 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Thu, 22 Jul 2021 05:04:50 +0000 Subject: [PATCH 018/762] pull: fix handling of multiple heads With multiple heads, we should not allow rebasing or fast-forwarding. Make sure any fast-forward request calls out specifically the fact that multiple branches are in play. Also, since we cannot fast-forward to multiple branches, fix our computation of can_ff. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- builtin/pull.c | 18 ++++++++++++++---- t/t7601-merge-pull-config.sh | 2 +- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/builtin/pull.c b/builtin/pull.c index 2f1d1f4037..b311ea6b9d 100644 --- a/builtin/pull.c +++ b/builtin/pull.c @@ -913,12 +913,18 @@ static int run_rebase(const struct object_id *newbase, return ret; } -static int get_can_ff(struct object_id *orig_head, struct object_id *orig_merge_head) +static int get_can_ff(struct object_id *orig_head, + struct oid_array *merge_heads) { int ret; struct commit_list *list = NULL; struct commit *merge_head, *head; + struct object_id *orig_merge_head; + if (merge_heads->nr > 1) + return 0; + + orig_merge_head = &merge_heads->oid[0]; head = lookup_commit_reference(the_repository, orig_head); commit_list_insert(head, &list); merge_head = lookup_commit_reference(the_repository, orig_merge_head); @@ -1057,10 +1063,14 @@ int cmd_pull(int argc, const char **argv, const char *prefix) die(_("Cannot merge multiple branches into empty head.")); return pull_into_void(merge_heads.oid, &curr_head); } - if (opt_rebase && merge_heads.nr > 1) - die(_("Cannot rebase onto multiple branches.")); + if (merge_heads.nr > 1) { + if (opt_rebase) + die(_("Cannot rebase onto multiple branches.")); + if (opt_ff && !strcmp(opt_ff, "--ff-only")) + die(_("Cannot fast-forward to multiple branches.")); + } - can_ff = get_can_ff(&orig_head, &merge_heads.oid[0]); + can_ff = get_can_ff(&orig_head, &merge_heads); /* ff-only takes precedence over rebase */ if (opt_ff && !strcmp(opt_ff, "--ff-only")) { diff --git a/t/t7601-merge-pull-config.sh b/t/t7601-merge-pull-config.sh index 742ed3981c..1f652f433e 100755 --- a/t/t7601-merge-pull-config.sh +++ b/t/t7601-merge-pull-config.sh @@ -331,7 +331,7 @@ test_expect_success 'Multiple heads warns about inability to fast forward' ' test_i18ngrep "You have divergent branches" err ' -test_expect_failure 'Multiple can never be fast forwarded' ' +test_expect_success 'Multiple can never be fast forwarded' ' git reset --hard c0 && test_must_fail git -c pull.ff=only pull . c1 c2 c3 2>err && test_i18ngrep ! "You have divergent branches" err && From b7e6a4162207785c66a1de6f4530499925b762b6 Mon Sep 17 00:00:00 2001 From: Emily Shaffer Date: Wed, 21 Jul 2021 18:27:06 -0700 Subject: [PATCH 019/762] tr2: make process info collection platform-generic To pave the way for non-Windows platforms to define trace2_collect_process_info(), reorganize the stub-or-definition schema to something which doesn't directly reference Windows. Platforms which want to collect parent process information in the future should: 1. Add an implementation to compat/ (e.g. compat/somearch/procinfo.c) 2. Add that object to COMPAT_OBJS to config.mak.uname (e.g. COMPAT_OBJS += compat/somearch/procinfo.o) 3. Define HAVE_PLATFORM_PROCINFO in config.mak.uname In the Windows case, this definition lives in compat/win32/trace2_win32_process_info.c, which is already conditionally added to COMPAT_OBJS; so let's add HAVE_PLATFORM_PROCINFO to hint to the build that compat/stub/procinfo.c should not be used. Signed-off-by: Emily Shaffer Helped-by: Jonathan Nieder Signed-off-by: Junio C Hamano --- Makefile | 4 ++++ compat/stub/procinfo.c | 11 +++++++++++ config.mak.uname | 1 + trace2.h | 6 ------ 4 files changed, 16 insertions(+), 6 deletions(-) create mode 100644 compat/stub/procinfo.c diff --git a/Makefile b/Makefile index 79bf9fe611..453cad4ee5 100644 --- a/Makefile +++ b/Makefile @@ -1890,6 +1890,10 @@ ifneq ($(PROCFS_EXECUTABLE_PATH),) BASIC_CFLAGS += '-DPROCFS_EXECUTABLE_PATH="$(procfs_executable_path_SQ)"' endif +ifndef HAVE_PLATFORM_PROCINFO + COMPAT_OBJS += compat/stub/procinfo.o +endif + ifdef HAVE_NS_GET_EXECUTABLE_PATH BASIC_CFLAGS += -DHAVE_NS_GET_EXECUTABLE_PATH endif diff --git a/compat/stub/procinfo.c b/compat/stub/procinfo.c new file mode 100644 index 0000000000..12c0a23c9e --- /dev/null +++ b/compat/stub/procinfo.c @@ -0,0 +1,11 @@ +#include "git-compat-util.h" + +#include "trace2.h" + +/* + * Stub. See sample implementations in compat/linux/procinfo.c and + * compat/win32/trace2_win32_process_info.c. + */ +void trace2_collect_process_info(enum trace2_process_info_reason reason) +{ +} diff --git a/config.mak.uname b/config.mak.uname index cb443b4e02..185ff79b14 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -612,6 +612,7 @@ ifneq (,$(findstring MINGW,$(uname_S))) ETAGS_TARGET = ETAGS NO_POSIX_GOODIES = UnfortunatelyYes DEFAULT_HELP_FORMAT = html + HAVE_PLATFORM_PROCINFO = YesPlease BASIC_LDFLAGS += -municode COMPAT_CFLAGS += -DNOGDI -Icompat -Icompat/win32 COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\" diff --git a/trace2.h b/trace2.h index ede18c2e06..0d990db817 100644 --- a/trace2.h +++ b/trace2.h @@ -492,13 +492,7 @@ enum trace2_process_info_reason { TRACE2_PROCESS_INFO_EXIT, }; -#if defined(GIT_WINDOWS_NATIVE) void trace2_collect_process_info(enum trace2_process_info_reason reason); -#else -#define trace2_collect_process_info(reason) \ - do { \ - } while (0) -#endif const char *trace2_session_id(void); From 2f732bf15e6dc9c2caf210784f180c6c059c570a Mon Sep 17 00:00:00 2001 From: Emily Shaffer Date: Wed, 21 Jul 2021 18:27:07 -0700 Subject: [PATCH 020/762] tr2: log parent process name It can be useful to tell who invoked Git - was it invoked manually by a user via CLI or script? By an IDE? In some cases - like 'repo' tool - we can influence the source code and set the GIT_TRACE2_PARENT_SID environment variable from the caller process. In 'repo''s case, that parent SID is manipulated to include the string "repo", which means we can positively identify when Git was invoked by 'repo' tool. However, identifying parents that way requires both that we know which tools invoke Git and that we have the ability to modify the source code of those tools. It cannot scale to keep up with the various IDEs and wrappers which use Git, most of which we don't know about. Learning which tools and wrappers invoke Git, and how, would give us insight to decide where to improve Git's usability and performance. Unfortunately, there's no cross-platform reliable way to gather the name of the parent process. If procfs is present, we can use that; otherwise we will need to discover the name another way. However, the process ID should be sufficient to look up the process name on most platforms, so that code may be shareable. Git for Windows gathers similar information and logs it as a "data_json" event. However, since "data_json" has a variable format, it is difficult to parse effectively in some languages; instead, let's pursue a dedicated "cmd_ancestry" event to record information about the ancestry of the current process and a consistent, parseable way. Git for Windows also gathers information about more than one generation of parent. In Linux further ancestry info can be gathered with procfs, but it's unwieldy to do so. In the interest of later moving Git for Windows ancestry logging to the 'cmd_ancestry' event, and in the interest of later adding more ancestry to the Linux implementation - or of adding this functionality to other platforms which have an easier time walking the process tree - let's make 'cmd_ancestry' accept an array of parentage. Signed-off-by: Emily Shaffer Signed-off-by: Junio C Hamano --- Documentation/technical/api-trace2.txt | 14 +++++++ compat/linux/procinfo.c | 55 ++++++++++++++++++++++++++ config.mak.uname | 2 + t/t0210/scrub_normal.perl | 6 +++ t/t0211/scrub_perf.perl | 5 +++ t/t0212/parse_events.perl | 5 ++- trace2.c | 13 ++++++ trace2.h | 10 +++++ trace2/tr2_tgt.h | 3 ++ trace2/tr2_tgt_event.c | 21 ++++++++++ trace2/tr2_tgt_normal.c | 19 +++++++++ trace2/tr2_tgt_perf.c | 16 ++++++++ 12 files changed, 168 insertions(+), 1 deletion(-) create mode 100644 compat/linux/procinfo.c diff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt index 3f52f981a2..8a0b360a0e 100644 --- a/Documentation/technical/api-trace2.txt +++ b/Documentation/technical/api-trace2.txt @@ -493,6 +493,20 @@ about specific error arguments. } ------------ +`"cmd_ancestry"`:: + This event contains the text command name for the parent (and earlier + generations of parents) of the current process, in an array ordered from + nearest parent to furthest great-grandparent. It may not be implemented + on all platforms. ++ +------------ +{ + "event":"cmd_ancestry", + ... + "ancestry":["bash","tmux: server","systemd"] +} +------------ + `"cmd_name"`:: This event contains the command name for this git process and the hierarchy of commands from parent git processes. diff --git a/compat/linux/procinfo.c b/compat/linux/procinfo.c new file mode 100644 index 0000000000..578fed4cd3 --- /dev/null +++ b/compat/linux/procinfo.c @@ -0,0 +1,55 @@ +#include "cache.h" + +#include "strbuf.h" +#include "strvec.h" +#include "trace2.h" + +static void get_ancestry_names(struct strvec *names) +{ + /* + * NEEDSWORK: We could gather the entire pstree into an array to match + * functionality with compat/win32/trace2_win32_process_info.c. + * To do so, we may want to examine /proc//stat. For now, just + * gather the immediate parent name which is readily accessible from + * /proc/$(getppid())/comm. + */ + struct strbuf procfs_path = STRBUF_INIT; + struct strbuf name = STRBUF_INIT; + + /* try to use procfs if it's present. */ + strbuf_addf(&procfs_path, "/proc/%d/comm", getppid()); + if (strbuf_read_file(&name, procfs_path.buf, 0)) { + strbuf_release(&procfs_path); + strbuf_trim_trailing_newline(&name); + strvec_push(names, strbuf_detach(&name, NULL)); + } + + return; + /* NEEDSWORK: add non-procfs-linux implementations here */ +} + +void trace2_collect_process_info(enum trace2_process_info_reason reason) +{ + if (!trace2_is_enabled()) + return; + + /* someday we may want to write something extra here, but not today */ + if (reason == TRACE2_PROCESS_INFO_EXIT) + return; + + if (reason == TRACE2_PROCESS_INFO_STARTUP) { + /* + * NEEDSWORK: we could do the entire ptree in an array instead, + * see compat/win32/trace2_win32_process_info.c. + */ + struct strvec names = STRVEC_INIT; + + get_ancestry_names(&names); + + if (names.nr) + trace2_cmd_ancestry(names.v); + strvec_clear(&names); + } + + return; +} diff --git a/config.mak.uname b/config.mak.uname index 185ff79b14..d3bd4c6843 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -58,6 +58,8 @@ ifeq ($(uname_S),Linux) FREAD_READS_DIRECTORIES = UnfortunatelyYes BASIC_CFLAGS += -DHAVE_SYSINFO PROCFS_EXECUTABLE_PATH = /proc/self/exe + HAVE_PLATFORM_PROCINFO = YesPlease + COMPAT_OBJS += compat/linux/procinfo.o endif ifeq ($(uname_S),GNU/kFreeBSD) HAVE_ALLOCA_H = YesPlease diff --git a/t/t0210/scrub_normal.perl b/t/t0210/scrub_normal.perl index c65d1a815e..7cc4de392a 100644 --- a/t/t0210/scrub_normal.perl +++ b/t/t0210/scrub_normal.perl @@ -42,6 +42,12 @@ while (<>) { # so just omit it for testing purposes. # print "cmd_path _EXE_\n"; } + elsif ($line =~ m/^cmd_ancestry/) { + # 'cmd_ancestry' is not implemented everywhere, so for portability's + # sake, skip it when parsing normal. + # + # print "$line"; + } else { print "$line"; } diff --git a/t/t0211/scrub_perf.perl b/t/t0211/scrub_perf.perl index 351af7844e..d164b750ff 100644 --- a/t/t0211/scrub_perf.perl +++ b/t/t0211/scrub_perf.perl @@ -44,6 +44,11 @@ while (<>) { # $tokens[$col_rest] = "_EXE_"; goto SKIP_LINE; } + elsif ($tokens[$col_event] =~ m/cmd_ancestry/) { + # 'cmd_ancestry' is platform-specific and not implemented everywhere, + # so skip it. + goto SKIP_LINE; + } elsif ($tokens[$col_event] =~ m/child_exit/) { $tokens[$col_rest] =~ s/ pid:\d* / pid:_PID_ /; } diff --git a/t/t0212/parse_events.perl b/t/t0212/parse_events.perl index 6584bb5634..b6408560c0 100644 --- a/t/t0212/parse_events.perl +++ b/t/t0212/parse_events.perl @@ -132,7 +132,10 @@ while (<>) { # just omit it for testing purposes. # $processes->{$sid}->{'path'} = "_EXE_"; } - + elsif ($event eq 'cmd_ancestry') { + # 'cmd_ancestry' is platform-specific and not implemented everywhere, so + # just skip it for testing purposes. + } elsif ($event eq 'cmd_name') { $processes->{$sid}->{'name'} = $line->{'name'}; $processes->{$sid}->{'hierarchy'} = $line->{'hierarchy'}; diff --git a/trace2.c b/trace2.c index 256120c7fd..b9b154ac44 100644 --- a/trace2.c +++ b/trace2.c @@ -260,6 +260,19 @@ void trace2_cmd_path_fl(const char *file, int line, const char *pathname) tgt_j->pfn_command_path_fl(file, line, pathname); } +void trace2_cmd_ancestry_fl(const char *file, int line, const char **parent_names) +{ + struct tr2_tgt *tgt_j; + int j; + + if (!trace2_enabled) + return; + + for_each_wanted_builtin (j, tgt_j) + if (tgt_j->pfn_command_ancestry_fl) + tgt_j->pfn_command_ancestry_fl(file, line, parent_names); +} + void trace2_cmd_name_fl(const char *file, int line, const char *name) { struct tr2_tgt *tgt_j; diff --git a/trace2.h b/trace2.h index 0d990db817..9b7286c572 100644 --- a/trace2.h +++ b/trace2.h @@ -133,6 +133,16 @@ void trace2_cmd_path_fl(const char *file, int line, const char *pathname); #define trace2_cmd_path(p) trace2_cmd_path_fl(__FILE__, __LINE__, (p)) +/* + * Emit an 'ancestry' event with the process name of the current process's + * parent process. + * This gives post-processors a way to determine what invoked the command and + * learn more about usage patterns. + */ +void trace2_cmd_ancestry_fl(const char *file, int line, const char **parent_names); + +#define trace2_cmd_ancestry(v) trace2_cmd_ancestry_fl(__FILE__, __LINE__, (v)) + /* * Emit a 'cmd_name' event with the canonical name of the command. * This gives post-processors a simple field to identify the command diff --git a/trace2/tr2_tgt.h b/trace2/tr2_tgt.h index 7b90469212..1f66fd6573 100644 --- a/trace2/tr2_tgt.h +++ b/trace2/tr2_tgt.h @@ -27,6 +27,8 @@ typedef void(tr2_tgt_evt_error_va_fl_t)(const char *file, int line, typedef void(tr2_tgt_evt_command_path_fl_t)(const char *file, int line, const char *command_path); +typedef void(tr2_tgt_evt_command_ancestry_fl_t)(const char *file, int line, + const char **parent_names); typedef void(tr2_tgt_evt_command_name_fl_t)(const char *file, int line, const char *name, const char *hierarchy); @@ -108,6 +110,7 @@ struct tr2_tgt { tr2_tgt_evt_atexit_t *pfn_atexit; tr2_tgt_evt_error_va_fl_t *pfn_error_va_fl; tr2_tgt_evt_command_path_fl_t *pfn_command_path_fl; + tr2_tgt_evt_command_ancestry_fl_t *pfn_command_ancestry_fl; tr2_tgt_evt_command_name_fl_t *pfn_command_name_fl; tr2_tgt_evt_command_mode_fl_t *pfn_command_mode_fl; tr2_tgt_evt_alias_fl_t *pfn_alias_fl; diff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c index 6353e8ad91..578a9a5287 100644 --- a/trace2/tr2_tgt_event.c +++ b/trace2/tr2_tgt_event.c @@ -261,6 +261,26 @@ static void fn_command_path_fl(const char *file, int line, const char *pathname) jw_release(&jw); } +static void fn_command_ancestry_fl(const char *file, int line, const char **parent_names) +{ + const char *event_name = "cmd_ancestry"; + const char *parent_name = NULL; + struct json_writer jw = JSON_WRITER_INIT; + + jw_object_begin(&jw, 0); + event_fmt_prepare(event_name, file, line, NULL, &jw); + jw_object_inline_begin_array(&jw, "ancestry"); + + while ((parent_name = *parent_names++)) + jw_array_string(&jw, parent_name); + + jw_end(&jw); /* 'ancestry' array */ + jw_end(&jw); /* event object */ + + tr2_dst_write_line(&tr2dst_event, &jw.json); + jw_release(&jw); +} + static void fn_command_name_fl(const char *file, int line, const char *name, const char *hierarchy) { @@ -584,6 +604,7 @@ struct tr2_tgt tr2_tgt_event = { fn_atexit, fn_error_va_fl, fn_command_path_fl, + fn_command_ancestry_fl, fn_command_name_fl, fn_command_mode_fl, fn_alias_fl, diff --git a/trace2/tr2_tgt_normal.c b/trace2/tr2_tgt_normal.c index 31b602c171..a5751c8864 100644 --- a/trace2/tr2_tgt_normal.c +++ b/trace2/tr2_tgt_normal.c @@ -160,6 +160,24 @@ static void fn_command_path_fl(const char *file, int line, const char *pathname) strbuf_release(&buf_payload); } +static void fn_command_ancestry_fl(const char *file, int line, const char **parent_names) +{ + const char *parent_name = NULL; + struct strbuf buf_payload = STRBUF_INIT; + + /* cmd_ancestry parent <- grandparent <- great-grandparent */ + strbuf_addstr(&buf_payload, "cmd_ancestry "); + while ((parent_name = *parent_names++)) { + strbuf_addstr(&buf_payload, parent_name); + /* if we'll write another one after this, add a delimiter */ + if (parent_names && *parent_names) + strbuf_addstr(&buf_payload, " <- "); + } + + normal_io_write_fl(file, line, &buf_payload); + strbuf_release(&buf_payload); +} + static void fn_command_name_fl(const char *file, int line, const char *name, const char *hierarchy) { @@ -306,6 +324,7 @@ struct tr2_tgt tr2_tgt_normal = { fn_atexit, fn_error_va_fl, fn_command_path_fl, + fn_command_ancestry_fl, fn_command_name_fl, fn_command_mode_fl, fn_alias_fl, diff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c index a8018f18cc..af4d65a0a5 100644 --- a/trace2/tr2_tgt_perf.c +++ b/trace2/tr2_tgt_perf.c @@ -253,6 +253,21 @@ static void fn_command_path_fl(const char *file, int line, const char *pathname) strbuf_release(&buf_payload); } +static void fn_command_ancestry_fl(const char *file, int line, const char **parent_names) +{ + const char *event_name = "cmd_ancestry"; + struct strbuf buf_payload = STRBUF_INIT; + + strbuf_addstr(&buf_payload, "ancestry:["); + /* It's not an argv but the rules are basically the same. */ + sq_append_quote_argv_pretty(&buf_payload, parent_names); + strbuf_addch(&buf_payload, ']'); + + perf_io_write_fl(file, line, event_name, NULL, NULL, NULL, NULL, + &buf_payload); + strbuf_release(&buf_payload); +} + static void fn_command_name_fl(const char *file, int line, const char *name, const char *hierarchy) { @@ -532,6 +547,7 @@ struct tr2_tgt tr2_tgt_perf = { fn_atexit, fn_error_va_fl, fn_command_path_fl, + fn_command_ancestry_fl, fn_command_name_fl, fn_command_mode_fl, fn_alias_fl, From 311d0b8e8e370bc9f6168af1f4e120d2b0975702 Mon Sep 17 00:00:00 2001 From: ZheNing Hu Date: Mon, 26 Jul 2021 03:26:46 +0000 Subject: [PATCH 021/762] ref-filter: add obj-type check in grab contents Only tag and commit objects use `grab_sub_body_contents()` to grab object contents in the current codebase. We want to teach the function to also handle blobs and trees to get their raw data, without parsing a blob (whose contents looks like a commit or a tag) incorrectly as a commit or a tag. So it's needed to pass a `struct expand_data *data` instread of only `void *buf` to both `grab_sub_body_contents()` and `grab_values()` to be able to check the object type. Skip the block of code that is specific to handling commits and tags early when the given object is of a wrong type to help later addition to handle other types of objects in this function. Reviewed-by: Jacob Keller Mentored-by: Christian Couder Mentored-by: Hariom Verma Helped-by: Junio C Hamano Signed-off-by: ZheNing Hu Signed-off-by: Junio C Hamano --- ref-filter.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/ref-filter.c b/ref-filter.c index 4db0e40ff4..5cee6512fb 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -1356,11 +1356,12 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size } /* See grab_values */ -static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf) +static void grab_sub_body_contents(struct atom_value *val, int deref, struct expand_data *data) { int i; const char *subpos = NULL, *bodypos = NULL, *sigpos = NULL; size_t sublen = 0, bodylen = 0, nonsiglen = 0, siglen = 0; + void *buf = data->content; for (i = 0; i < used_atom_cnt; i++) { struct used_atom *atom = &used_atom[i]; @@ -1371,10 +1372,13 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf) continue; if (deref) name++; - if (strcmp(name, "body") && - !starts_with(name, "subject") && - !starts_with(name, "trailers") && - !starts_with(name, "contents")) + + if ((data->type != OBJ_TAG && + data->type != OBJ_COMMIT) || + (strcmp(name, "body") && + !starts_with(name, "subject") && + !starts_with(name, "trailers") && + !starts_with(name, "contents"))) continue; if (!subpos) find_subpos(buf, @@ -1438,17 +1442,19 @@ static void fill_missing_values(struct atom_value *val) * pointed at by the ref itself; otherwise it is the object the * ref (which is a tag) refers to. */ -static void grab_values(struct atom_value *val, int deref, struct object *obj, void *buf) +static void grab_values(struct atom_value *val, int deref, struct object *obj, struct expand_data *data) { + void *buf = data->content; + switch (obj->type) { case OBJ_TAG: grab_tag_values(val, deref, obj); - grab_sub_body_contents(val, deref, buf); + grab_sub_body_contents(val, deref, data); grab_person("tagger", val, deref, buf); break; case OBJ_COMMIT: grab_commit_values(val, deref, obj); - grab_sub_body_contents(val, deref, buf); + grab_sub_body_contents(val, deref, data); grab_person("author", val, deref, buf); grab_person("committer", val, deref, buf); break; @@ -1678,7 +1684,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj return strbuf_addf_ret(err, -1, _("parse_object_buffer failed on %s for %s"), oid_to_hex(&oi->oid), ref->refname); } - grab_values(ref->value, deref, *obj, oi->content); + grab_values(ref->value, deref, *obj, oi); } grab_common_values(ref->value, deref, oi); From bd0708c7eb8be43242180a1ea2a0f0acda7a4fbc Mon Sep 17 00:00:00 2001 From: ZheNing Hu Date: Mon, 26 Jul 2021 03:26:47 +0000 Subject: [PATCH 022/762] ref-filter: add %(raw) atom MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add new formatting option `%(raw)`, which will print the raw object data without any changes. It will help further to migrate all cat-file formatting logic from cat-file to ref-filter. The raw data of blob, tree objects may contain '\0', but most of the logic in `ref-filter` depends on the output of the atom being text (specifically, no embedded NULs in it). E.g. `quote_formatting()` use `strbuf_addstr()` or `*._quote_buf()` add the data to the buffer. The raw data of a tree object is `100644 one\0...`, only the `100644 one` will be added to the buffer, which is incorrect. Therefore, we need to find a way to record the length of the atom_value's member `s`. Although strbuf can already record the string and its length, if we want to replace the type of atom_value's member `s` with strbuf, many places in ref-filter that are filled with dynamically allocated mermory in `v->s` are not easy to replace. At the same time, we need to check if `v->s == NULL` in populate_value(), and strbuf cannot easily distinguish NULL and empty strings, but c-style "const char *" can do it. So add a new member in `struct atom_value`: `s_size`, which can record raw object size, it can help us add raw object data to the buffer or compare two buffers which contain raw object data. Note that `--format=%(raw)` cannot be used with `--python`, `--shell`, `--tcl`, and `--perl` because if the binary raw data is passed to a variable in such languages, these may not support arbitrary binary data in their string variable type. Reviewed-by: Jacob Keller Mentored-by: Christian Couder Mentored-by: Hariom Verma Helped-by: Bagas Sanjaya Helped-by: Ævar Arnfjörð Bjarmason Helped-by: Felipe Contreras Helped-by: Phillip Wood Helped-by: Junio C Hamano Based-on-patch-by: Olga Telezhnaya Signed-off-by: ZheNing Hu Signed-off-by: Junio C Hamano --- Documentation/git-for-each-ref.txt | 9 ++ ref-filter.c | 143 +++++++++++++++---- t/t6300-for-each-ref.sh | 216 +++++++++++++++++++++++++++++ 3 files changed, 340 insertions(+), 28 deletions(-) diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt index 2ae2478de7..cbb6f87d13 100644 --- a/Documentation/git-for-each-ref.txt +++ b/Documentation/git-for-each-ref.txt @@ -235,6 +235,15 @@ and `date` to extract the named component. For email fields (`authoremail`, without angle brackets, and `:localpart` to get the part before the `@` symbol out of the trimmed email. +The raw data in an object is `raw`. + +raw:size:: + The raw data size of the object. + +Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`, +`--perl` because such language may not support arbitrary binary data in their +string variable type. + The message in a commit or a tag object is `contents`, from which `contents:` can be used to extract various parts out of: diff --git a/ref-filter.c b/ref-filter.c index 5cee6512fb..0d5eb91ed5 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -144,6 +144,7 @@ enum atom_type { ATOM_BODY, ATOM_TRAILERS, ATOM_CONTENTS, + ATOM_RAW, ATOM_UPSTREAM, ATOM_PUSH, ATOM_SYMREF, @@ -189,6 +190,9 @@ static struct used_atom { struct process_trailer_options trailer_opts; unsigned int nlines; } contents; + struct { + enum { RAW_BARE, RAW_LENGTH } option; + } raw_data; struct { cmp_status cmp_status; const char *str; @@ -426,6 +430,18 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato return 0; } +static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom, + const char *arg, struct strbuf *err) +{ + if (!arg) + atom->u.raw_data.option = RAW_BARE; + else if (!strcmp(arg, "size")) + atom->u.raw_data.option = RAW_LENGTH; + else + return strbuf_addf_ret(err, -1, _("unrecognized %%(raw) argument: %s"), arg); + return 0; +} + static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { @@ -586,6 +602,7 @@ static struct { [ATOM_BODY] = { "body", SOURCE_OBJ, FIELD_STR, body_atom_parser }, [ATOM_TRAILERS] = { "trailers", SOURCE_OBJ, FIELD_STR, trailers_atom_parser }, [ATOM_CONTENTS] = { "contents", SOURCE_OBJ, FIELD_STR, contents_atom_parser }, + [ATOM_RAW] = { "raw", SOURCE_OBJ, FIELD_STR, raw_atom_parser }, [ATOM_UPSTREAM] = { "upstream", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser }, [ATOM_PUSH] = { "push", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser }, [ATOM_SYMREF] = { "symref", SOURCE_NONE, FIELD_STR, refname_atom_parser }, @@ -620,12 +637,19 @@ struct ref_formatting_state { struct atom_value { const char *s; + ssize_t s_size; int (*handler)(struct atom_value *atomv, struct ref_formatting_state *state, struct strbuf *err); uintmax_t value; /* used for sorting when not FIELD_STR */ struct used_atom *atom; }; +#define ATOM_SIZE_UNSPECIFIED (-1) + +#define ATOM_VALUE_INIT { \ + .s_size = ATOM_SIZE_UNSPECIFIED \ +} + /* * Used to parse format string and sort specifiers */ @@ -644,13 +668,6 @@ static int parse_ref_filter_atom(const struct ref_format *format, return strbuf_addf_ret(err, -1, _("malformed field name: %.*s"), (int)(ep-atom), atom); - /* Do we have the atom already used elsewhere? */ - for (i = 0; i < used_atom_cnt; i++) { - int len = strlen(used_atom[i].name); - if (len == ep - atom && !memcmp(used_atom[i].name, atom, len)) - return i; - } - /* * If the atom name has a colon, strip it and everything after * it off - it specifies the format for this entry, and @@ -660,6 +677,13 @@ static int parse_ref_filter_atom(const struct ref_format *format, arg = memchr(sp, ':', ep - sp); atom_len = (arg ? arg : ep) - sp; + /* Do we have the atom already used elsewhere? */ + for (i = 0; i < used_atom_cnt; i++) { + int len = strlen(used_atom[i].name); + if (len == ep - atom && !memcmp(used_atom[i].name, atom, len)) + return i; + } + /* Is the atom a valid one? */ for (i = 0; i < ARRAY_SIZE(valid_atom); i++) { int len = strlen(valid_atom[i].name); @@ -709,11 +733,14 @@ static int parse_ref_filter_atom(const struct ref_format *format, return at; } -static void quote_formatting(struct strbuf *s, const char *str, int quote_style) +static void quote_formatting(struct strbuf *s, const char *str, ssize_t len, int quote_style) { switch (quote_style) { case QUOTE_NONE: - strbuf_addstr(s, str); + if (len < 0) + strbuf_addstr(s, str); + else + strbuf_add(s, str, len); break; case QUOTE_SHELL: sq_quote_buf(s, str); @@ -740,9 +767,11 @@ static int append_atom(struct atom_value *v, struct ref_formatting_state *state, * encountered. */ if (!state->stack->prev) - quote_formatting(&state->stack->output, v->s, state->quote_style); - else + quote_formatting(&state->stack->output, v->s, v->s_size, state->quote_style); + else if (v->s_size < 0) strbuf_addstr(&state->stack->output, v->s); + else + strbuf_add(&state->stack->output, v->s, v->s_size); return 0; } @@ -842,21 +871,23 @@ static int if_atom_handler(struct atom_value *atomv, struct ref_formatting_state return 0; } -static int is_empty(const char *s) +static int is_empty(struct strbuf *buf) { - while (*s != '\0') { - if (!isspace(*s)) - return 0; - s++; - } - return 1; -} + const char *cur = buf->buf; + const char *end = buf->buf + buf->len; + + while (cur != end && (isspace(*cur))) + cur++; + + return cur == end; + } static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_state *state, struct strbuf *err) { struct ref_formatting_stack *cur = state->stack; struct if_then_else *if_then_else = NULL; + size_t str_len = 0; if (cur->at_end == if_then_else_handler) if_then_else = (struct if_then_else *)cur->at_end_data; @@ -867,18 +898,22 @@ static int then_atom_handler(struct atom_value *atomv, struct ref_formatting_sta if (if_then_else->else_atom_seen) return strbuf_addf_ret(err, -1, _("format: %%(then) atom used after %%(else)")); if_then_else->then_atom_seen = 1; + if (if_then_else->str) + str_len = strlen(if_then_else->str); /* * If the 'equals' or 'notequals' attribute is used then * perform the required comparison. If not, only non-empty * strings satisfy the 'if' condition. */ if (if_then_else->cmp_status == COMPARE_EQUAL) { - if (!strcmp(if_then_else->str, cur->output.buf)) + if (str_len == cur->output.len && + !memcmp(if_then_else->str, cur->output.buf, cur->output.len)) if_then_else->condition_satisfied = 1; } else if (if_then_else->cmp_status == COMPARE_UNEQUAL) { - if (strcmp(if_then_else->str, cur->output.buf)) + if (str_len != cur->output.len || + memcmp(if_then_else->str, cur->output.buf, cur->output.len)) if_then_else->condition_satisfied = 1; - } else if (cur->output.len && !is_empty(cur->output.buf)) + } else if (cur->output.len && !is_empty(&cur->output)) if_then_else->condition_satisfied = 1; strbuf_reset(&cur->output); return 0; @@ -924,7 +959,7 @@ static int end_atom_handler(struct atom_value *atomv, struct ref_formatting_stat * only on the topmost supporting atom. */ if (!current->prev->prev) { - quote_formatting(&s, current->output.buf, state->quote_style); + quote_formatting(&s, current->output.buf, current->output.len, state->quote_style); strbuf_swap(¤t->output, &s); } strbuf_release(&s); @@ -974,6 +1009,10 @@ int verify_ref_format(struct ref_format *format) at = parse_ref_filter_atom(format, sp + 2, ep, &err); if (at < 0) die("%s", err.buf); + if (format->quote_style && used_atom[at].atom_type == ATOM_RAW && + used_atom[at].u.raw_data.option == RAW_BARE) + die(_("--format=%.*s cannot be used with" + "--python, --shell, --tcl, --perl"), (int)(ep - sp - 2), sp + 2); cp = ep + 1; if (skip_prefix(used_atom[at].name, "color:", &color)) @@ -1367,12 +1406,25 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp struct used_atom *atom = &used_atom[i]; const char *name = atom->name; struct atom_value *v = &val[i]; + enum atom_type atom_type = atom->atom_type; if (!!deref != (*name == '*')) continue; if (deref) name++; + if (atom_type == ATOM_RAW) { + unsigned long buf_size = data->size; + + if (atom->u.raw_data.option == RAW_BARE) { + v->s = xmemdupz(buf, buf_size); + v->s_size = buf_size; + } else if (atom->u.raw_data.option == RAW_LENGTH) { + v->s = xstrfmt("%"PRIuMAX, (uintmax_t)buf_size); + } + continue; + } + if ((data->type != OBJ_TAG && data->type != OBJ_COMMIT) || (strcmp(name, "body") && @@ -1460,9 +1512,11 @@ static void grab_values(struct atom_value *val, int deref, struct object *obj, s break; case OBJ_TREE: /* grab_tree_values(val, deref, obj, buf, sz); */ + grab_sub_body_contents(val, deref, data); break; case OBJ_BLOB: /* grab_blob_values(val, deref, obj, buf, sz); */ + grab_sub_body_contents(val, deref, data); break; default: die("Eh? Object of type %d?", obj->type); @@ -1766,6 +1820,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) const char *refname; struct branch *branch = NULL; + v->s_size = ATOM_SIZE_UNSPECIFIED; v->handler = append_atom; v->atom = atom; @@ -2369,6 +2424,19 @@ static int compare_detached_head(struct ref_array_item *a, struct ref_array_item return 0; } +static int memcasecmp(const void *vs1, const void *vs2, size_t n) +{ + const char *s1 = vs1, *s2 = vs2; + const char *end = s1 + n; + + for (; s1 < end; s1++, s2++) { + int diff = tolower(*s1) - tolower(*s2); + if (diff) + return diff; + } + return 0; +} + static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, struct ref_array_item *b) { struct atom_value *va, *vb; @@ -2389,10 +2457,29 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru } else if (s->sort_flags & REF_SORTING_VERSION) { cmp = versioncmp(va->s, vb->s); } else if (cmp_type == FIELD_STR) { - int (*cmp_fn)(const char *, const char *); - cmp_fn = s->sort_flags & REF_SORTING_ICASE - ? strcasecmp : strcmp; - cmp = cmp_fn(va->s, vb->s); + if (va->s_size < 0 && vb->s_size < 0) { + int (*cmp_fn)(const char *, const char *); + cmp_fn = s->sort_flags & REF_SORTING_ICASE + ? strcasecmp : strcmp; + cmp = cmp_fn(va->s, vb->s); + } else { + size_t a_size = va->s_size < 0 ? + strlen(va->s) : va->s_size; + size_t b_size = vb->s_size < 0 ? + strlen(vb->s) : vb->s_size; + int (*cmp_fn)(const void *, const void *, size_t); + cmp_fn = s->sort_flags & REF_SORTING_ICASE + ? memcasecmp : memcmp; + + cmp = cmp_fn(va->s, vb->s, b_size > a_size ? + a_size : b_size); + if (!cmp) { + if (a_size > b_size) + cmp = 1; + else if (a_size < b_size) + cmp = -1; + } + } } else { if (va->value < vb->value) cmp = -1; @@ -2491,7 +2578,7 @@ int format_ref_array_item(struct ref_array_item *info, append_literal(cp, sp, &state); } if (format->need_color_reset_at_eol) { - struct atom_value resetv; + struct atom_value resetv = ATOM_VALUE_INIT; resetv.s = GIT_COLOR_RESET; if (append_atom(&resetv, &state, error_buf)) { pop_stack_element(&state.stack); diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh index 9e0214076b..18554f62d9 100755 --- a/t/t6300-for-each-ref.sh +++ b/t/t6300-for-each-ref.sh @@ -130,6 +130,8 @@ test_atom head parent:short=10 '' test_atom head numparent 0 test_atom head object '' test_atom head type '' +test_atom head raw "$(git cat-file commit refs/heads/main) +" test_atom head '*objectname' '' test_atom head '*objecttype' '' test_atom head author 'A U Thor 1151968724 +0200' @@ -221,6 +223,15 @@ test_atom tag contents 'Tagging at 1151968727 ' test_atom tag HEAD ' ' +test_expect_success 'basic atom: refs/tags/testtag *raw' ' + git cat-file commit refs/tags/testtag^{} >expected && + git for-each-ref --format="%(*raw)" refs/tags/testtag >actual && + sanitize_pgp expected.clean && + echo >>expected.clean && + sanitize_pgp actual.clean && + test_cmp expected.clean actual.clean +' + test_expect_success 'Check invalid atoms names are errors' ' test_must_fail git for-each-ref --format="%(INVALID)" refs/heads ' @@ -686,6 +697,15 @@ test_atom refs/tags/signed-empty contents:body '' test_atom refs/tags/signed-empty contents:signature "$sig" test_atom refs/tags/signed-empty contents "$sig" +test_expect_success GPG 'basic atom: refs/tags/signed-empty raw' ' + git cat-file tag refs/tags/signed-empty >expected && + git for-each-ref --format="%(raw)" refs/tags/signed-empty >actual && + sanitize_pgp expected.clean && + echo >>expected.clean && + sanitize_pgp actual.clean && + test_cmp expected.clean actual.clean +' + test_atom refs/tags/signed-short subject 'subject line' test_atom refs/tags/signed-short subject:sanitize 'subject-line' test_atom refs/tags/signed-short contents:subject 'subject line' @@ -695,6 +715,15 @@ test_atom refs/tags/signed-short contents:signature "$sig" test_atom refs/tags/signed-short contents "subject line $sig" +test_expect_success GPG 'basic atom: refs/tags/signed-short raw' ' + git cat-file tag refs/tags/signed-short >expected && + git for-each-ref --format="%(raw)" refs/tags/signed-short >actual && + sanitize_pgp expected.clean && + echo >>expected.clean && + sanitize_pgp actual.clean && + test_cmp expected.clean actual.clean +' + test_atom refs/tags/signed-long subject 'subject line' test_atom refs/tags/signed-long subject:sanitize 'subject-line' test_atom refs/tags/signed-long contents:subject 'subject line' @@ -708,6 +737,15 @@ test_atom refs/tags/signed-long contents "subject line body contents $sig" +test_expect_success GPG 'basic atom: refs/tags/signed-long raw' ' + git cat-file tag refs/tags/signed-long >expected && + git for-each-ref --format="%(raw)" refs/tags/signed-long >actual && + sanitize_pgp expected.clean && + echo >>expected.clean && + sanitize_pgp actual.clean && + test_cmp expected.clean actual.clean +' + test_expect_success 'set up refs pointing to tree and blob' ' git update-ref refs/mytrees/first refs/heads/main^{tree} && git update-ref refs/myblobs/first refs/heads/main:one @@ -720,6 +758,16 @@ test_atom refs/mytrees/first contents:body "" test_atom refs/mytrees/first contents:signature "" test_atom refs/mytrees/first contents "" +test_expect_success 'basic atom: refs/mytrees/first raw' ' + git cat-file tree refs/mytrees/first >expected && + echo >>expected && + git for-each-ref --format="%(raw)" refs/mytrees/first >actual && + test_cmp expected actual && + git cat-file -s refs/mytrees/first >expected && + git for-each-ref --format="%(raw:size)" refs/mytrees/first >actual && + test_cmp expected actual +' + test_atom refs/myblobs/first subject "" test_atom refs/myblobs/first contents:subject "" test_atom refs/myblobs/first body "" @@ -727,6 +775,174 @@ test_atom refs/myblobs/first contents:body "" test_atom refs/myblobs/first contents:signature "" test_atom refs/myblobs/first contents "" +test_expect_success 'basic atom: refs/myblobs/first raw' ' + git cat-file blob refs/myblobs/first >expected && + echo >>expected && + git for-each-ref --format="%(raw)" refs/myblobs/first >actual && + test_cmp expected actual && + git cat-file -s refs/myblobs/first >expected && + git for-each-ref --format="%(raw:size)" refs/myblobs/first >actual && + test_cmp expected actual +' + +test_expect_success 'set up refs pointing to binary blob' ' + printf "a\0b\0c" >blob1 && + printf "a\0c\0b" >blob2 && + printf "\0a\0b\0c" >blob3 && + printf "abc" >blob4 && + printf "\0 \0 \0 " >blob5 && + printf "\0 \0a\0 " >blob6 && + printf " " >blob7 && + >blob8 && + obj=$(git hash-object -w blob1) && + git update-ref refs/myblobs/blob1 "$obj" && + obj=$(git hash-object -w blob2) && + git update-ref refs/myblobs/blob2 "$obj" && + obj=$(git hash-object -w blob3) && + git update-ref refs/myblobs/blob3 "$obj" && + obj=$(git hash-object -w blob4) && + git update-ref refs/myblobs/blob4 "$obj" && + obj=$(git hash-object -w blob5) && + git update-ref refs/myblobs/blob5 "$obj" && + obj=$(git hash-object -w blob6) && + git update-ref refs/myblobs/blob6 "$obj" && + obj=$(git hash-object -w blob7) && + git update-ref refs/myblobs/blob7 "$obj" && + obj=$(git hash-object -w blob8) && + git update-ref refs/myblobs/blob8 "$obj" +' + +test_expect_success 'Verify sorts with raw' ' + cat >expected <<-EOF && + refs/myblobs/blob8 + refs/myblobs/blob5 + refs/myblobs/blob6 + refs/myblobs/blob3 + refs/myblobs/blob7 + refs/mytrees/first + refs/myblobs/first + refs/myblobs/blob1 + refs/myblobs/blob2 + refs/myblobs/blob4 + refs/heads/main + EOF + git for-each-ref --format="%(refname)" --sort=raw \ + refs/heads/main refs/myblobs/ refs/mytrees/first >actual && + test_cmp expected actual +' + +test_expect_success 'Verify sorts with raw:size' ' + cat >expected <<-EOF && + refs/myblobs/blob8 + refs/myblobs/first + refs/myblobs/blob7 + refs/heads/main + refs/myblobs/blob4 + refs/myblobs/blob1 + refs/myblobs/blob2 + refs/myblobs/blob3 + refs/myblobs/blob5 + refs/myblobs/blob6 + refs/mytrees/first + EOF + git for-each-ref --format="%(refname)" --sort=raw:size \ + refs/heads/main refs/myblobs/ refs/mytrees/first >actual && + test_cmp expected actual +' + +test_expect_success 'validate raw atom with %(if:equals)' ' + cat >expected <<-EOF && + not equals + not equals + not equals + not equals + not equals + not equals + refs/myblobs/blob4 + not equals + not equals + not equals + not equals + not equals + EOF + git for-each-ref --format="%(if:equals=abc)%(raw)%(then)%(refname)%(else)not equals%(end)" \ + refs/myblobs/ refs/heads/ >actual && + test_cmp expected actual +' + +test_expect_success 'validate raw atom with %(if:notequals)' ' + cat >expected <<-EOF && + refs/heads/ambiguous + refs/heads/main + refs/heads/newtag + refs/myblobs/blob1 + refs/myblobs/blob2 + refs/myblobs/blob3 + equals + refs/myblobs/blob5 + refs/myblobs/blob6 + refs/myblobs/blob7 + refs/myblobs/blob8 + refs/myblobs/first + EOF + git for-each-ref --format="%(if:notequals=abc)%(raw)%(then)%(refname)%(else)equals%(end)" \ + refs/myblobs/ refs/heads/ >actual && + test_cmp expected actual +' + +test_expect_success 'empty raw refs with %(if)' ' + cat >expected <<-EOF && + refs/myblobs/blob1 not empty + refs/myblobs/blob2 not empty + refs/myblobs/blob3 not empty + refs/myblobs/blob4 not empty + refs/myblobs/blob5 not empty + refs/myblobs/blob6 not empty + refs/myblobs/blob7 empty + refs/myblobs/blob8 empty + refs/myblobs/first not empty + EOF + git for-each-ref --format="%(refname) %(if)%(raw)%(then)not empty%(else)empty%(end)" \ + refs/myblobs/ >actual && + test_cmp expected actual +' + +test_expect_success '%(raw) with --python must fail' ' + test_must_fail git for-each-ref --format="%(raw)" --python +' + +test_expect_success '%(raw) with --tcl must fail' ' + test_must_fail git for-each-ref --format="%(raw)" --tcl +' + +test_expect_success '%(raw) with --perl must fail' ' + test_must_fail git for-each-ref --format="%(raw)" --perl +' + +test_expect_success '%(raw) with --shell must fail' ' + test_must_fail git for-each-ref --format="%(raw)" --shell +' + +test_expect_success '%(raw) with --shell and --sort=raw must fail' ' + test_must_fail git for-each-ref --format="%(raw)" --sort=raw --shell +' + +test_expect_success '%(raw:size) with --shell' ' + git for-each-ref --format="%(raw:size)" | while read line + do + echo "'\''$line'\''" >>expect + done && + git for-each-ref --format="%(raw:size)" --shell >actual && + test_cmp expect actual +' + +test_expect_success 'for-each-ref --format compare with cat-file --batch' ' + git rev-parse refs/mytrees/first | git cat-file --batch >expected && + git for-each-ref --format="%(objectname) %(objecttype) %(objectsize) +%(raw)" refs/mytrees/first >actual && + test_cmp expected actual +' + test_expect_success 'set up multiple-sort tags' ' for when in 100000 200000 do From 7121c4d4e2877115fb372b3f147fad4cd1306751 Mon Sep 17 00:00:00 2001 From: ZheNing Hu Date: Mon, 26 Jul 2021 03:26:48 +0000 Subject: [PATCH 023/762] ref-filter: --format=%(raw) support --perl MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Because the perl language can handle binary data correctly, add the function perl_quote_buf_with_len(), which can specify the length of the data and prevent the data from being truncated at '\0' to help `--format="%(raw)"` support `--perl`. Reviewed-by: Jacob Keller Helped-by: Ævar Arnfjörð Bjarmason Signed-off-by: ZheNing Hu Signed-off-by: Junio C Hamano --- Documentation/git-for-each-ref.txt | 4 ++-- quote.c | 17 +++++++++++++++++ quote.h | 1 + ref-filter.c | 15 +++++++++++---- t/t6300-for-each-ref.sh | 19 +++++++++++++++++-- 5 files changed, 48 insertions(+), 8 deletions(-) diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt index cbb6f87d13..6da899c629 100644 --- a/Documentation/git-for-each-ref.txt +++ b/Documentation/git-for-each-ref.txt @@ -241,8 +241,8 @@ raw:size:: The raw data size of the object. Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`, -`--perl` because such language may not support arbitrary binary data in their -string variable type. +because such language may not support arbitrary binary data in their string +variable type. The message in a commit or a tag object is `contents`, from which `contents:` can be used to extract various parts out of: diff --git a/quote.c b/quote.c index 8a3a5e39eb..26719d21d1 100644 --- a/quote.c +++ b/quote.c @@ -471,6 +471,23 @@ void perl_quote_buf(struct strbuf *sb, const char *src) strbuf_addch(sb, sq); } +void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len) +{ + const char sq = '\''; + const char bq = '\\'; + const char *c = src; + const char *end = src + len; + + strbuf_addch(sb, sq); + while (c != end) { + if (*c == sq || *c == bq) + strbuf_addch(sb, bq); + strbuf_addch(sb, *c); + c++; + } + strbuf_addch(sb, sq); +} + void python_quote_buf(struct strbuf *sb, const char *src) { const char sq = '\''; diff --git a/quote.h b/quote.h index 768cc6338e..0fe69e264b 100644 --- a/quote.h +++ b/quote.h @@ -94,6 +94,7 @@ char *quote_path(const char *in, const char *prefix, struct strbuf *out, unsigne /* quoting as a string literal for other languages */ void perl_quote_buf(struct strbuf *sb, const char *src); +void perl_quote_buf_with_len(struct strbuf *sb, const char *src, size_t len); void python_quote_buf(struct strbuf *sb, const char *src); void tcl_quote_buf(struct strbuf *sb, const char *src); void basic_regex_quote_buf(struct strbuf *sb, const char *src); diff --git a/ref-filter.c b/ref-filter.c index 0d5eb91ed5..ce6c211501 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -746,7 +746,10 @@ static void quote_formatting(struct strbuf *s, const char *str, ssize_t len, int sq_quote_buf(s, str); break; case QUOTE_PERL: - perl_quote_buf(s, str); + if (len < 0) + perl_quote_buf(s, str); + else + perl_quote_buf_with_len(s, str, len); break; case QUOTE_PYTHON: python_quote_buf(s, str); @@ -1009,10 +1012,14 @@ int verify_ref_format(struct ref_format *format) at = parse_ref_filter_atom(format, sp + 2, ep, &err); if (at < 0) die("%s", err.buf); - if (format->quote_style && used_atom[at].atom_type == ATOM_RAW && - used_atom[at].u.raw_data.option == RAW_BARE) + + if ((format->quote_style == QUOTE_PYTHON || + format->quote_style == QUOTE_SHELL || + format->quote_style == QUOTE_TCL) && + used_atom[at].atom_type == ATOM_RAW && + used_atom[at].u.raw_data.option == RAW_BARE) die(_("--format=%.*s cannot be used with" - "--python, --shell, --tcl, --perl"), (int)(ep - sp - 2), sp + 2); + "--python, --shell, --tcl"), (int)(ep - sp - 2), sp + 2); cp = ep + 1; if (skip_prefix(used_atom[at].name, "color:", &color)) diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh index 18554f62d9..3d15d0a536 100755 --- a/t/t6300-for-each-ref.sh +++ b/t/t6300-for-each-ref.sh @@ -915,8 +915,23 @@ test_expect_success '%(raw) with --tcl must fail' ' test_must_fail git for-each-ref --format="%(raw)" --tcl ' -test_expect_success '%(raw) with --perl must fail' ' - test_must_fail git for-each-ref --format="%(raw)" --perl +test_expect_success '%(raw) with --perl' ' + git for-each-ref --format="\$name= %(raw); +print \"\$name\"" refs/myblobs/blob1 --perl | perl >actual && + cmp blob1 actual && + git for-each-ref --format="\$name= %(raw); +print \"\$name\"" refs/myblobs/blob3 --perl | perl >actual && + cmp blob3 actual && + git for-each-ref --format="\$name= %(raw); +print \"\$name\"" refs/myblobs/blob8 --perl | perl >actual && + cmp blob8 actual && + git for-each-ref --format="\$name= %(raw); +print \"\$name\"" refs/myblobs/first --perl | perl >actual && + cmp one actual && + git cat-file tree refs/mytrees/first > expected && + git for-each-ref --format="\$name= %(raw); +print \"\$name\"" refs/mytrees/first --perl | perl >actual && + cmp expected actual ' test_expect_success '%(raw) with --shell must fail' ' From e85fcb355a38b5f01493efecdf0150a10ee471da Mon Sep 17 00:00:00 2001 From: ZheNing Hu Date: Mon, 26 Jul 2021 03:26:49 +0000 Subject: [PATCH 024/762] ref-filter: use non-const ref_format in *_atom_parser() Use non-const ref_format in *_atom_parser(), which can help us modify the members of ref_format in *_atom_parser(). Mentored-by: Christian Couder Mentored-by: Hariom Verma Signed-off-by: ZheNing Hu Signed-off-by: Junio C Hamano --- builtin/tag.c | 2 +- ref-filter.c | 44 ++++++++++++++++++++++---------------------- ref-filter.h | 4 ++-- 3 files changed, 25 insertions(+), 25 deletions(-) diff --git a/builtin/tag.c b/builtin/tag.c index 82fcfc0982..452558ec95 100644 --- a/builtin/tag.c +++ b/builtin/tag.c @@ -146,7 +146,7 @@ static int verify_tag(const char *name, const char *ref, const struct object_id *oid, void *cb_data) { int flags; - const struct ref_format *format = cb_data; + struct ref_format *format = cb_data; flags = GPG_VERIFY_VERBOSE; if (format->format) diff --git a/ref-filter.c b/ref-filter.c index ce6c211501..f944160c5c 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -226,7 +226,7 @@ static int strbuf_addf_ret(struct strbuf *sb, int ret, const char *fmt, ...) return ret; } -static int color_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int color_atom_parser(struct ref_format *format, struct used_atom *atom, const char *color_value, struct strbuf *err) { if (!color_value) @@ -264,7 +264,7 @@ static int refname_atom_parser_internal(struct refname_atom *atom, const char *a return 0; } -static int remote_ref_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int remote_ref_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { struct string_list params = STRING_LIST_INIT_DUP; @@ -311,7 +311,7 @@ static int remote_ref_atom_parser(const struct ref_format *format, struct used_a return 0; } -static int objecttype_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int objecttype_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { if (arg) @@ -323,7 +323,7 @@ static int objecttype_atom_parser(const struct ref_format *format, struct used_a return 0; } -static int objectsize_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int objectsize_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { if (!arg) { @@ -343,7 +343,7 @@ static int objectsize_atom_parser(const struct ref_format *format, struct used_a return 0; } -static int deltabase_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int deltabase_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { if (arg) @@ -355,7 +355,7 @@ static int deltabase_atom_parser(const struct ref_format *format, struct used_at return 0; } -static int body_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int body_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { if (arg) @@ -364,7 +364,7 @@ static int body_atom_parser(const struct ref_format *format, struct used_atom *a return 0; } -static int subject_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int subject_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { if (!arg) @@ -376,7 +376,7 @@ static int subject_atom_parser(const struct ref_format *format, struct used_atom return 0; } -static int trailers_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int trailers_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { atom->u.contents.trailer_opts.no_divider = 1; @@ -402,7 +402,7 @@ static int trailers_atom_parser(const struct ref_format *format, struct used_ato return 0; } -static int contents_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int contents_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { if (!arg) @@ -430,7 +430,7 @@ static int contents_atom_parser(const struct ref_format *format, struct used_ato return 0; } -static int raw_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int raw_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { if (!arg) @@ -442,7 +442,7 @@ static int raw_atom_parser(const struct ref_format *format, struct used_atom *at return 0; } -static int oid_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int oid_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { if (!arg) @@ -461,7 +461,7 @@ static int oid_atom_parser(const struct ref_format *format, struct used_atom *at return 0; } -static int person_email_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int person_email_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { if (!arg) @@ -475,7 +475,7 @@ static int person_email_atom_parser(const struct ref_format *format, struct used return 0; } -static int refname_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int refname_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { return refname_atom_parser_internal(&atom->u.refname, arg, atom->name, err); @@ -492,7 +492,7 @@ static align_type parse_align_position(const char *s) return -1; } -static int align_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int align_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { struct align *align = &atom->u.align; @@ -544,7 +544,7 @@ static int align_atom_parser(const struct ref_format *format, struct used_atom * return 0; } -static int if_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int if_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err) { if (!arg) { @@ -559,7 +559,7 @@ static int if_atom_parser(const struct ref_format *format, struct used_atom *ato return 0; } -static int head_atom_parser(const struct ref_format *format, struct used_atom *atom, +static int head_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *unused_err) { atom->u.head = resolve_refdup("HEAD", RESOLVE_REF_READING, NULL, NULL); @@ -570,7 +570,7 @@ static struct { const char *name; info_source source; cmp_type cmp_type; - int (*parser)(const struct ref_format *format, struct used_atom *atom, + int (*parser)(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *err); } valid_atom[] = { [ATOM_REFNAME] = { "refname", SOURCE_NONE, FIELD_STR, refname_atom_parser }, @@ -653,7 +653,7 @@ struct atom_value { /* * Used to parse format string and sort specifiers */ -static int parse_ref_filter_atom(const struct ref_format *format, +static int parse_ref_filter_atom(struct ref_format *format, const char *atom, const char *ep, struct strbuf *err) { @@ -2556,9 +2556,9 @@ static void append_literal(const char *cp, const char *ep, struct ref_formatting } int format_ref_array_item(struct ref_array_item *info, - const struct ref_format *format, - struct strbuf *final_buf, - struct strbuf *error_buf) + struct ref_format *format, + struct strbuf *final_buf, + struct strbuf *error_buf) { const char *cp, *sp, *ep; struct ref_formatting_state state = REF_FORMATTING_STATE_INIT; @@ -2602,7 +2602,7 @@ int format_ref_array_item(struct ref_array_item *info, } void pretty_print_ref(const char *name, const struct object_id *oid, - const struct ref_format *format) + struct ref_format *format) { struct ref_array_item *ref_item; struct strbuf output = STRBUF_INIT; diff --git a/ref-filter.h b/ref-filter.h index baf72a7189..74fb423fc8 100644 --- a/ref-filter.h +++ b/ref-filter.h @@ -116,7 +116,7 @@ void ref_array_sort(struct ref_sorting *sort, struct ref_array *array); void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting, unsigned int mask, int on); /* Based on the given format and quote_style, fill the strbuf */ int format_ref_array_item(struct ref_array_item *info, - const struct ref_format *format, + struct ref_format *format, struct strbuf *final_buf, struct strbuf *error_buf); /* Parse a single sort specifier and add it to the list */ @@ -137,7 +137,7 @@ void setup_ref_filter_porcelain_msg(void); * name must be a fully qualified refname. */ void pretty_print_ref(const char *name, const struct object_id *oid, - const struct ref_format *format); + struct ref_format *format); /* * Push a single ref onto the array; this can be used to construct your own From b9dee075eb07713a49922f7e1f90a6a9de5c5e5f Mon Sep 17 00:00:00 2001 From: ZheNing Hu Date: Mon, 26 Jul 2021 03:26:50 +0000 Subject: [PATCH 025/762] ref-filter: add %(rest) atom %(rest) is a atom used for cat-file batch mode, which can split the input lines at the first whitespace boundary, all characters before that whitespace are considered to be the object name; characters after that first run of whitespace (i.e., the "rest" of the line) are output in place of the %(rest) atom. In order to let "cat-file --batch=%(rest)" use the ref-filter interface, add %(rest) atom for ref-filter. Introduce the reject_atom() to reject the atom %(rest) for "git for-each-ref", "git branch", "git tag" and "git verify-tag". Reviewed-by: Jacob Keller Suggected-by: Jacob Keller Mentored-by: Christian Couder Mentored-by: Hariom Verma Signed-off-by: ZheNing Hu Signed-off-by: Junio C Hamano --- ref-filter.c | 25 +++++++++++++++++++++++++ ref-filter.h | 5 ++++- t/t3203-branch-output.sh | 4 ++++ t/t6300-for-each-ref.sh | 4 ++++ t/t7004-tag.sh | 4 ++++ t/t7030-verify-tag.sh | 4 ++++ 6 files changed, 45 insertions(+), 1 deletion(-) diff --git a/ref-filter.c b/ref-filter.c index f944160c5c..2cacee9e64 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -157,6 +157,7 @@ enum atom_type { ATOM_IF, ATOM_THEN, ATOM_ELSE, + ATOM_REST, }; /* @@ -559,6 +560,15 @@ static int if_atom_parser(struct ref_format *format, struct used_atom *atom, return 0; } +static int rest_atom_parser(struct ref_format *format, struct used_atom *atom, + const char *arg, struct strbuf *err) +{ + if (arg) + return strbuf_addf_ret(err, -1, _("%%(rest) does not take arguments")); + format->use_rest = 1; + return 0; +} + static int head_atom_parser(struct ref_format *format, struct used_atom *atom, const char *arg, struct strbuf *unused_err) { @@ -615,6 +625,7 @@ static struct { [ATOM_IF] = { "if", SOURCE_NONE, FIELD_STR, if_atom_parser }, [ATOM_THEN] = { "then", SOURCE_NONE }, [ATOM_ELSE] = { "else", SOURCE_NONE }, + [ATOM_REST] = { "rest", SOURCE_NONE, FIELD_STR, rest_atom_parser }, /* * Please update $__git_ref_fieldlist in git-completion.bash * when you add new atoms @@ -992,6 +1003,11 @@ static const char *find_next(const char *cp) return NULL; } +static int reject_atom(enum atom_type atom_type) +{ + return atom_type == ATOM_REST; +} + /* * Make sure the format string is well formed, and parse out * the used atoms. @@ -1012,6 +1028,8 @@ int verify_ref_format(struct ref_format *format) at = parse_ref_filter_atom(format, sp + 2, ep, &err); if (at < 0) die("%s", err.buf); + if (reject_atom(used_atom[at].atom_type)) + die(_("this command reject atom %%(%.*s)"), (int)(ep - sp - 2), sp + 2); if ((format->quote_style == QUOTE_PYTHON || format->quote_style == QUOTE_SHELL || @@ -1931,6 +1949,12 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) v->handler = else_atom_handler; v->s = xstrdup(""); continue; + } else if (atom_type == ATOM_REST) { + if (ref->rest) + v->s = xstrdup(ref->rest); + else + v->s = xstrdup(""); + continue; } else continue; @@ -2148,6 +2172,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname, FLEX_ALLOC_STR(ref, refname, refname); oidcpy(&ref->objectname, oid); + ref->rest = NULL; return ref; } diff --git a/ref-filter.h b/ref-filter.h index 74fb423fc8..c15dee8d6b 100644 --- a/ref-filter.h +++ b/ref-filter.h @@ -38,6 +38,7 @@ struct ref_sorting { struct ref_array_item { struct object_id objectname; + const char *rest; int flag; unsigned int kind; const char *symref; @@ -76,14 +77,16 @@ struct ref_format { * verify_ref_format() afterwards to finalize. */ const char *format; + const char *rest; int quote_style; + int use_rest; int use_color; /* Internal state to ref-filter */ int need_color_reset_at_eol; }; -#define REF_FORMAT_INIT { NULL, 0, -1 } +#define REF_FORMAT_INIT { .use_color = -1 } /* Macros for checking --merged and --no-merged options */ #define _OPT_MERGED_NO_MERGED(option, filter, h) \ diff --git a/t/t3203-branch-output.sh b/t/t3203-branch-output.sh index 5325b9f67a..6e94c6db7b 100755 --- a/t/t3203-branch-output.sh +++ b/t/t3203-branch-output.sh @@ -340,6 +340,10 @@ test_expect_success 'git branch --format option' ' test_cmp expect actual ' +test_expect_success 'git branch with --format=%(rest) must fail' ' + test_must_fail git branch --format="%(rest)" >actual +' + test_expect_success 'worktree colors correct' ' cat >expect <<-EOF && * (HEAD detached from fromtag) diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh index 3d15d0a536..0d2e062f79 100755 --- a/t/t6300-for-each-ref.sh +++ b/t/t6300-for-each-ref.sh @@ -1211,6 +1211,10 @@ test_expect_success 'basic atom: head contents:trailers' ' test_cmp expect actual.clean ' +test_expect_success 'basic atom: rest must fail' ' + test_must_fail git for-each-ref --format="%(rest)" refs/heads/main +' + test_expect_success 'trailer parsing not fooled by --- line' ' git commit --allow-empty -F - <<-\EOF && this is the subject diff --git a/t/t7004-tag.sh b/t/t7004-tag.sh index 2f72c5c688..082be85dff 100755 --- a/t/t7004-tag.sh +++ b/t/t7004-tag.sh @@ -1998,6 +1998,10 @@ test_expect_success '--format should list tags as per format given' ' test_cmp expect actual ' +test_expect_success 'git tag -l with --format="%(rest)" must fail' ' + test_must_fail git tag -l --format="%(rest)" "v1*" +' + test_expect_success "set up color tests" ' echo "v1.0" >expect.color && echo "v1.0" >expect.bare && diff --git a/t/t7030-verify-tag.sh b/t/t7030-verify-tag.sh index 3cefde9602..10faa64515 100755 --- a/t/t7030-verify-tag.sh +++ b/t/t7030-verify-tag.sh @@ -194,6 +194,10 @@ test_expect_success GPG 'verifying tag with --format' ' test_cmp expect actual ' +test_expect_success GPG 'verifying tag with --format="%(rest)" must fail' ' + test_must_fail git verify-tag --format="%(rest)" "fourth-signed" +' + test_expect_success GPG 'verifying a forged tag with --format should fail silently' ' test_must_fail git verify-tag --format="tagname : %(tag)" $(cat forged1.tag) >actual-forged && test_must_be_empty actual-forged From b7d11a0f5d2db1eff623150cdc2a2ddccd6845b3 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin Date: Sat, 24 Jul 2021 22:06:49 +0000 Subject: [PATCH 026/762] tests: exercise the RUNTIME_PREFIX feature Originally, we refrained from adding a regression test in 7b6c6496374 (system_path(): Add prefix computation at runtime if RUNTIME_PREFIX set, 2008-08-10), and in 226c0ddd0d6 (exec_cmd: RUNTIME_PREFIX on some POSIX systems, 2018-04-10). The reason was that it was deemed too tricky to test. Turns out that it is not tricky to test at all: we simply create a pseudo-root, copy the `git` executable into the `git/` subdirectory of that pseudo-root, then copy a script into the `libexec/git-core/` directory and expect that to be picked up. As long as the trash directory is in a location where binaries can be executed, this works. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano --- Makefile | 5 +++++ t/t0060-path-utils.sh | 18 ++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/Makefile b/Makefile index c3565fc0f8..4c3e4ccabc 100644 --- a/Makefile +++ b/Makefile @@ -2826,6 +2826,11 @@ ifdef GIT_TEST_INDEX_VERSION endif ifdef GIT_TEST_PERL_FATAL_WARNINGS @echo GIT_TEST_PERL_FATAL_WARNINGS=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_PERL_FATAL_WARNINGS)))'\' >>$@+ +endif +ifdef RUNTIME_PREFIX + @echo RUNTIME_PREFIX=\'true\' >>$@+ +else + @echo RUNTIME_PREFIX=\'false\' >>$@+ endif @if cmp $@+ $@ >/dev/null 2>&1; then $(RM) $@+; else mv $@+ $@; fi diff --git a/t/t0060-path-utils.sh b/t/t0060-path-utils.sh index de4960783f..a76728c27b 100755 --- a/t/t0060-path-utils.sh +++ b/t/t0060-path-utils.sh @@ -525,4 +525,22 @@ test_expect_success MINGW 'is_valid_path() on Windows' ' "PRN./abc" ' +test_lazy_prereq RUNTIME_PREFIX ' + test true = "$RUNTIME_PREFIX" +' + +test_lazy_prereq CAN_EXEC_IN_PWD ' + cp "$GIT_EXEC_PATH"/git$X ./ && + ./git rev-parse +' + +test_expect_success RUNTIME_PREFIX,CAN_EXEC_IN_PWD 'RUNTIME_PREFIX works' ' + mkdir -p pretend/bin pretend/libexec/git-core && + echo "echo HERE" | write_script pretend/libexec/git-core/git-here && + cp "$GIT_EXEC_PATH"/git$X pretend/bin/ && + GIT_EXEC_PATH= ./pretend/bin/git here >actual && + echo HERE >expect && + test_cmp expect actual +' + test_done From 789f6f226b6c7a5b28468044c1705c62aef5eeb9 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin Date: Sat, 24 Jul 2021 22:06:50 +0000 Subject: [PATCH 027/762] expand_user_path(): remove stale part of the comment In 395de250d9d (Expand ~ and ~user in core.excludesfile, commit.template, 2009-11-17), the `user_path()` function was refactored into the `expand_user_path()`. During that refactoring, the `buf` parameter was lost, but the code comment above said function still talks about it. Let's remove that stale part of the comment. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano --- path.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/path.c b/path.c index 7bccd830e9..3318ad2433 100644 --- a/path.c +++ b/path.c @@ -719,9 +719,8 @@ static struct passwd *getpw_str(const char *username, size_t len) } /* - * Return a string with ~ and ~user expanded via getpw*. If buf != NULL, - * then it is a newly allocated string. Returns NULL on getpw failure or - * if path is NULL. + * Return a string with ~ and ~user expanded via getpw*. Returns NULL on getpw + * failure or if path is NULL. * * If real_home is true, strbuf_realpath($HOME) is used in the expansion. */ From 644e6b2c0f55fa52de88586d375e03ac7f7914b7 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin Date: Sat, 24 Jul 2021 22:06:51 +0000 Subject: [PATCH 028/762] expand_user_path(): clarify the role of the `real_home` parameter The `real_home` parameter only has an effect when expanding paths starting with `~/`, not when expanding paths starting with `~/`. Let's make that clear. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano --- path.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/path.c b/path.c index 3318ad2433..bf329e535c 100644 --- a/path.c +++ b/path.c @@ -722,7 +722,7 @@ static struct passwd *getpw_str(const char *username, size_t len) * Return a string with ~ and ~user expanded via getpw*. Returns NULL on getpw * failure or if path is NULL. * - * If real_home is true, strbuf_realpath($HOME) is used in the expansion. + * If real_home is true, strbuf_realpath($HOME) is used in the `~/` expansion. */ char *expand_user_path(const char *path, int real_home) { From a03b097d6307763c6778f5a1f194fbcbd158a5f7 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin Date: Sat, 24 Jul 2021 22:06:52 +0000 Subject: [PATCH 029/762] Use a better name for the function interpolating paths It is not immediately clear what `expand_user_path()` means, so let's rename it to `interpolate_path()`. This also opens the path for interpolating more than just a home directory. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano --- builtin/credential-cache.c | 2 +- builtin/credential-store.c | 2 +- builtin/gc.c | 2 +- cache.h | 2 +- config.c | 8 ++++---- path.c | 4 ++-- sequencer.c | 2 +- 7 files changed, 11 insertions(+), 11 deletions(-) diff --git a/builtin/credential-cache.c b/builtin/credential-cache.c index 76a6ba3722..e8a7415747 100644 --- a/builtin/credential-cache.c +++ b/builtin/credential-cache.c @@ -90,7 +90,7 @@ static char *get_socket_path(void) { struct stat sb; char *old_dir, *socket; - old_dir = expand_user_path("~/.git-credential-cache", 0); + old_dir = interpolate_path("~/.git-credential-cache", 0); if (old_dir && !stat(old_dir, &sb) && S_ISDIR(sb.st_mode)) socket = xstrfmt("%s/socket", old_dir); else diff --git a/builtin/credential-store.c b/builtin/credential-store.c index ae3c1ba75f..62a4f3c265 100644 --- a/builtin/credential-store.c +++ b/builtin/credential-store.c @@ -173,7 +173,7 @@ int cmd_credential_store(int argc, const char **argv, const char *prefix) if (file) { string_list_append(&fns, file); } else { - if ((file = expand_user_path("~/.git-credentials", 0))) + if ((file = interpolate_path("~/.git-credentials", 0))) string_list_append_nodup(&fns, file); file = xdg_config_home("credentials"); if (file) diff --git a/builtin/gc.c b/builtin/gc.c index f05d2f0a1a..6ce5ca4512 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1542,7 +1542,7 @@ static char *launchctl_service_filename(const char *name) struct strbuf filename = STRBUF_INIT; strbuf_addf(&filename, "~/Library/LaunchAgents/%s.plist", name); - expanded = expand_user_path(filename.buf, 1); + expanded = interpolate_path(filename.buf, 1); if (!expanded) die(_("failed to expand path '%s'"), filename.buf); diff --git a/cache.h b/cache.h index ba04ff8bd3..87e4cbe9c5 100644 --- a/cache.h +++ b/cache.h @@ -1246,7 +1246,7 @@ typedef int create_file_fn(const char *path, void *cb); int raceproof_create_file(const char *path, create_file_fn fn, void *cb); int mkdir_in_gitdir(const char *path); -char *expand_user_path(const char *path, int real_home); +char *interpolate_path(const char *path, int real_home); const char *enter_repo(const char *path, int strict); static inline int is_absolute_path(const char *path) { diff --git a/config.c b/config.c index f9c400ad30..c17b979729 100644 --- a/config.c +++ b/config.c @@ -137,7 +137,7 @@ static int handle_path_include(const char *path, struct config_include_data *inc if (!path) return config_error_nonbool("include.path"); - expanded = expand_user_path(path, 0); + expanded = interpolate_path(path, 0); if (!expanded) return error(_("could not expand include path '%s'"), path); path = expanded; @@ -185,7 +185,7 @@ static int prepare_include_condition_pattern(struct strbuf *pat) char *expanded; int prefix = 0; - expanded = expand_user_path(pat->buf, 1); + expanded = interpolate_path(pat->buf, 1); if (expanded) { strbuf_reset(pat); strbuf_addstr(pat, expanded); @@ -1270,7 +1270,7 @@ int git_config_pathname(const char **dest, const char *var, const char *value) { if (!value) return config_error_nonbool(var); - *dest = expand_user_path(value, 0); + *dest = interpolate_path(value, 0); if (!*dest) die(_("failed to expand user dir in: '%s'"), value); return 0; @@ -1844,7 +1844,7 @@ void git_global_config(char **user_out, char **xdg_out) char *xdg_config = NULL; if (!user_config) { - user_config = expand_user_path("~/.gitconfig", 0); + user_config = interpolate_path("~/.gitconfig", 0); xdg_config = xdg_config_home("config"); } diff --git a/path.c b/path.c index bf329e535c..8dc5ad2cb5 100644 --- a/path.c +++ b/path.c @@ -724,7 +724,7 @@ static struct passwd *getpw_str(const char *username, size_t len) * * If real_home is true, strbuf_realpath($HOME) is used in the `~/` expansion. */ -char *expand_user_path(const char *path, int real_home) +char *interpolate_path(const char *path, int real_home) { struct strbuf user_path = STRBUF_INIT; const char *to_copy = path; @@ -811,7 +811,7 @@ const char *enter_repo(const char *path, int strict) strbuf_add(&validated_path, path, len); if (used_path.buf[0] == '~') { - char *newpath = expand_user_path(used_path.buf, 0); + char *newpath = interpolate_path(used_path.buf, 0); if (!newpath) return NULL; strbuf_attach(&used_path, newpath, strlen(newpath), diff --git a/sequencer.c b/sequencer.c index 0bec01cf38..007a851804 100644 --- a/sequencer.c +++ b/sequencer.c @@ -1241,7 +1241,7 @@ N_("Your name and email address were configured automatically based\n" static const char *implicit_ident_advice(void) { - char *user_config = expand_user_path("~/.gitconfig", 0); + char *user_config = interpolate_path("~/.gitconfig", 0); char *xdg_config = xdg_config_home("config"); int config_exists = file_exists(user_config) || file_exists(xdg_config); From e394a16023cbb62784e380f70ad8a833fb960d68 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin Date: Sat, 24 Jul 2021 22:06:53 +0000 Subject: [PATCH 030/762] interpolate_path(): allow specifying paths relative to the runtime prefix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ever since Git learned to detect its install location at runtime, there was the slightly awkward problem that it was impossible to specify paths relative to said location. For example, if a version of Git was shipped with custom SSL certificates to use, there was no portable way to specify `http.sslCAInfo`. In Git for Windows, the problem was "solved" for years by interpreting paths starting with a slash as relative to the runtime prefix. However, this is not correct: such paths _are_ legal on Windows, and they are interpreted as absolute paths in the same drive as the current directory. After a lengthy discussion, and an even lengthier time to mull over the problem and its best solution, and then more discussions, we eventually decided to introduce support for the magic sequence `%(prefix)/`. If a path starts with this, the remainder is interpreted as relative to the detected (runtime) prefix. If built without runtime prefix support, Git will simply interpolate the compiled-in prefix. If a user _wants_ to specify a path starting with the magic sequence, they can prefix the magic sequence with `./` and voilà, the path won't be expanded. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano --- Documentation/config.txt | 9 +++++++++ path.c | 8 ++++++++ t/t0060-path-utils.sh | 8 ++++++++ 3 files changed, 25 insertions(+) diff --git a/Documentation/config.txt b/Documentation/config.txt index bf82766a6a..0c0e6b859f 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -298,6 +298,15 @@ pathname:: tilde expansion happens to such a string: `~/` is expanded to the value of `$HOME`, and `~user/` to the specified user's home directory. ++ +If a path starts with `%(prefix)/`, the remainder is interpreted as a +path relative to Git's "runtime prefix", i.e. relative to the location +where Git itself was installed. For example, `%(prefix)/bin/` refers to +the directory in which the Git executable itself lives. If Git was +compiled without runtime prefix support, the compiled-in prefix will be +subsituted instead. In the unlikely event that a literal path needs to +be specified that should _not_ be expanded, it needs to be prefixed by +`./`, like so: `./%(prefix)/bin`. Variables diff --git a/path.c b/path.c index 8dc5ad2cb5..0bc788ea40 100644 --- a/path.c +++ b/path.c @@ -12,6 +12,7 @@ #include "packfile.h" #include "object-store.h" #include "lockfile.h" +#include "exec-cmd.h" static int get_st_mode_bits(const char *path, int *mode) { @@ -723,6 +724,9 @@ static struct passwd *getpw_str(const char *username, size_t len) * failure or if path is NULL. * * If real_home is true, strbuf_realpath($HOME) is used in the `~/` expansion. + * + * If the path starts with `%(prefix)/`, the remainder is interpreted as + * relative to where Git is installed, and expanded to the absolute path. */ char *interpolate_path(const char *path, int real_home) { @@ -731,6 +735,10 @@ char *interpolate_path(const char *path, int real_home) if (path == NULL) goto return_null; + + if (skip_prefix(path, "%(prefix)/", &path)) + return system_path(path); + if (path[0] == '~') { const char *first_slash = strchrnul(path, '/'); const char *username = path + 1; diff --git a/t/t0060-path-utils.sh b/t/t0060-path-utils.sh index a76728c27b..34d1061f32 100755 --- a/t/t0060-path-utils.sh +++ b/t/t0060-path-utils.sh @@ -540,6 +540,14 @@ test_expect_success RUNTIME_PREFIX,CAN_EXEC_IN_PWD 'RUNTIME_PREFIX works' ' cp "$GIT_EXEC_PATH"/git$X pretend/bin/ && GIT_EXEC_PATH= ./pretend/bin/git here >actual && echo HERE >expect && + test_cmp expect actual' + +test_expect_success RUNTIME_PREFIX,CAN_EXEC_IN_PWD '%(prefix)/ works' ' + mkdir -p pretend/bin && + cp "$GIT_EXEC_PATH"/git$X pretend/bin/ && + git config yes.path "%(prefix)/yes" && + GIT_EXEC_PATH= ./pretend/bin/git config --path yes.path >actual && + echo "$(pwd)/pretend/yes" >expect && test_cmp expect actual ' From 7ed37eb8ae485ff5590c656e871d6c739edfa067 Mon Sep 17 00:00:00 2001 From: Junio C Hamano Date: Mon, 26 Jul 2021 14:55:05 -0700 Subject: [PATCH 031/762] expand_user_path: allow in-flight topics to keep using the old name Signed-off-by: Junio C Hamano --- cache.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cache.h b/cache.h index 87e4cbe9c5..679a27e17c 100644 --- a/cache.h +++ b/cache.h @@ -1247,6 +1247,8 @@ int raceproof_create_file(const char *path, create_file_fn fn, void *cb); int mkdir_in_gitdir(const char *path); char *interpolate_path(const char *path, int real_home); +/* NEEDSWORK: remove this synonym once in-flight topics have migrated */ +#define expand_user_path interpolate_path const char *enter_repo(const char *path, int strict); static inline int is_absolute_path(const char *path) { From 4577d26dc0aaae3bb35c8906db96ba043716f3f7 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 26 Jul 2021 14:33:56 -0400 Subject: [PATCH 032/762] t4060: remove unused variable Signed-off-by: David Turner Signed-off-by: Junio C Hamano --- t/t4060-diff-submodule-option-diff-format.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/t/t4060-diff-submodule-option-diff-format.sh b/t/t4060-diff-submodule-option-diff-format.sh index dc7b242697..69b9946931 100755 --- a/t/t4060-diff-submodule-option-diff-format.sh +++ b/t/t4060-diff-submodule-option-diff-format.sh @@ -361,7 +361,6 @@ test_expect_success 'typechanged submodule(submodule->blob)' ' rm -f sm1 && test_create_repo sm1 && head6=$(add_file sm1 foo6 foo7) -fullhead6=$(cd sm1; git rev-parse --verify HEAD) test_expect_success 'nonexistent commit' ' git diff-index -p --submodule=diff HEAD >actual && cat >expected <<-EOF && From ffcb4e94d309db2889dac1d3dcbd0f4b2a0f6390 Mon Sep 17 00:00:00 2001 From: Junio C Hamano Date: Tue, 27 Jul 2021 11:22:18 -0700 Subject: [PATCH 033/762] bisect: do not run show-branch just to show the current commit In scripted versions of "git bisect", we used "git show-branch" to describe a single commit in the bisect log and also to the interactive user after checking out the next version to be tested. The former use of "git show-branch" was lost when the helper function that wrote bisect log entries was rewritten at 0f30233a (bisect--helper: `bisect_write` shell function in C, 2019-01-02) in C But we've kept the latter ever since 0871984d (bisect: make "git bisect" use new "--next-all" bisect-helper function, 2009-05-09) started using the faithful C-rewrite introduced at ef24c7ca (bisect--helper: add "--next-exit" to output bisect results, 2009-04-19). Showing "[] " is simple enough with our helper pretty.c::format_commit_message() and spawning show-branch is an overkill. Let's lose one external process. Signed-off-by: Junio C Hamano --- bisect.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/bisect.c b/bisect.c index af2863d044..2b8b6546e9 100644 --- a/bisect.c +++ b/bisect.c @@ -23,7 +23,6 @@ static struct oid_array skipped_revs; static struct object_id *current_bad_oid; static const char *argv_checkout[] = {"checkout", "-q", NULL, "--", NULL}; -static const char *argv_show_branch[] = {"show-branch", NULL, NULL}; static const char *term_bad; static const char *term_good; @@ -729,6 +728,9 @@ static enum bisect_error bisect_checkout(const struct object_id *bisect_rev, int { char bisect_rev_hex[GIT_MAX_HEXSZ + 1]; enum bisect_error res = BISECT_OK; + struct commit *commit; + struct pretty_print_context pp = {0}; + struct strbuf commit_msg = STRBUF_INIT; oid_to_hex_r(bisect_rev_hex, bisect_rev); update_ref(NULL, "BISECT_EXPECTED_REV", bisect_rev, NULL, 0, UPDATE_REFS_DIE_ON_ERR); @@ -748,13 +750,11 @@ static enum bisect_error bisect_checkout(const struct object_id *bisect_rev, int return -abs(res); } - argv_show_branch[1] = bisect_rev_hex; - res = run_command_v_opt(argv_show_branch, RUN_GIT_CMD); - /* - * Errors in `run_command()` itself, signaled by res < 0, - * and errors in the child process, signaled by res > 0 - * can both be treated as regular BISECT_FAILURE (-1). - */ + commit = lookup_commit_reference(the_repository, bisect_rev); + format_commit_message(commit, "[%H] %s%n", &commit_msg, &pp); + fputs(commit_msg.buf, stdout); + strbuf_release(&commit_msg); + return -abs(res); } From 1fcc40cd1dbfbdf6879a7b60a3a0250e46dc5c60 Mon Sep 17 00:00:00 2001 From: Junio C Hamano Date: Wed, 28 Jul 2021 10:07:02 -0700 Subject: [PATCH 034/762] bisect: simplify return code from bisect_checkout() The function was designed to return only BISECT_OK (0) or BISECT_FAILED (-1) and no other values, but there were two issues: - The comment misspelled BISECT_FAILED as BISECT_FAILURE, even though the logic it described (i.e. any non-zero return should be reported as a single BISECT_FAILED) was correct. - It took the return value from run_command_v_opt(), and assumed it was either -1 or 1 upon error, which is not the case; it can relay errors from wait_or_whine(), which can report exit status of the child process. Translate any error return from run_command_v_opt() to BISECT_FAILED, and simplify the resulting code by losing the 'res' variable that is no longer needed. Signed-off-by: Junio C Hamano --- bisect.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/bisect.c b/bisect.c index 2b8b6546e9..888949fba6 100644 --- a/bisect.c +++ b/bisect.c @@ -727,7 +727,6 @@ static int is_expected_rev(const struct object_id *oid) static enum bisect_error bisect_checkout(const struct object_id *bisect_rev, int no_checkout) { char bisect_rev_hex[GIT_MAX_HEXSZ + 1]; - enum bisect_error res = BISECT_OK; struct commit *commit; struct pretty_print_context pp = {0}; struct strbuf commit_msg = STRBUF_INIT; @@ -740,14 +739,13 @@ static enum bisect_error bisect_checkout(const struct object_id *bisect_rev, int update_ref(NULL, "BISECT_HEAD", bisect_rev, NULL, 0, UPDATE_REFS_DIE_ON_ERR); } else { - res = run_command_v_opt(argv_checkout, RUN_GIT_CMD); - if (res) + if (run_command_v_opt(argv_checkout, RUN_GIT_CMD)) /* * Errors in `run_command()` itself, signaled by res < 0, * and errors in the child process, signaled by res > 0 - * can both be treated as regular BISECT_FAILURE (-1). + * can both be treated as regular BISECT_FAILED (-1). */ - return -abs(res); + return BISECT_FAILED; } commit = lookup_commit_reference(the_repository, bisect_rev); @@ -755,7 +753,7 @@ static enum bisect_error bisect_checkout(const struct object_id *bisect_rev, int fputs(commit_msg.buf, stdout); strbuf_release(&commit_msg); - return -abs(res); + return BISECT_OK; } static struct commit *get_commit_reference(struct repository *r, From 83ad8ca596dd879d2907a240fb24a1603bb1decc Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 29 Jul 2021 14:52:03 +0000 Subject: [PATCH 035/762] t1092: test merge conflicts outside cone Conflicts can occur outside of the sparse-checkout definition, and in that case users might try to resolve the conflicts in several ways. Document a few of these ways in a test. Make it clear that this behavior is not necessarily the optimal flow, since users can become confused when Git deletes these files from the worktree in later commands. Reviewed-by: Elijah Newren Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- t/t1092-sparse-checkout-compatibility.sh | 43 ++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 91e30d6ec2..4c3bcb3499 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -114,6 +114,16 @@ test_expect_success 'setup' ' git add . && git commit -m "file to dir" && + for side in left right + do + git checkout -b merge-$side base && + echo $side >>deep/deeper2/a && + echo $side >>folder1/a && + echo $side >>folder2/a && + git add . && + git commit -m "$side" || return 1 + done && + git checkout -b deepest base && echo "updated deepest" >deep/deeper1/deepest/a && git commit -a -m "update deepest" && @@ -482,6 +492,39 @@ test_expect_success 'merge' ' test_all_match git rev-parse HEAD^{tree} ' +# NEEDSWORK: This test is documenting current behavior, but that +# behavior can be confusing to users so there is desire to change it. +# Right now, users might be using this flow to work through conflicts, +# so any solution should present advice to users who try this sequence +# of commands to follow whatever new method we create. +test_expect_success 'merge with conflict outside cone' ' + init_repos && + + test_all_match git checkout -b merge-tip merge-left && + test_all_match git status --porcelain=v2 && + test_all_match test_must_fail git merge -m merge merge-right && + test_all_match git status --porcelain=v2 && + + # Resolve the conflict in different ways: + # 1. Revert to the base + test_all_match git checkout base -- deep/deeper2/a && + test_all_match git status --porcelain=v2 && + + # 2. Add the file with conflict markers + test_all_match git add folder1/a && + test_all_match git status --porcelain=v2 && + + # 3. Rename the file to another sparse filename and + # accept conflict markers as resolved content. + run_on_all mv folder2/a folder2/z && + test_all_match git add folder2 && + test_all_match git status --porcelain=v2 && + + test_all_match git merge --continue && + test_all_match git status --porcelain=v2 && + test_all_match git rev-parse HEAD^{tree} +' + test_expect_success 'merge with outside renames' ' init_repos && From 5e7cbab19680745d4f3e0c50903025d9f02e7468 Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 29 Jul 2021 14:52:04 +0000 Subject: [PATCH 036/762] add: allow operating on a sparse-only index Disable command_requires_full_index for 'git add'. This does not require any additional removals of ensure_full_index(). The main reason is that 'git add' discovers changes based on the pathspec and the worktree itself. These are then inserted into the index directly, and calls to index_name_pos() or index_file_exists() already call expand_to_path() at the appropriate time to support a sparse-index. Add a test to check that 'git add -A' and 'git add ' does not expand the index at all, as long as is not within a sparse directory. This does not help the global 'git add .' case. We can measure the improvement using p2000-sparse-operations.sh with these results: Test HEAD~1 HEAD ------------------------------------------------------------------------------ 2000.6: git add -A (full-index-v3) 0.35(0.30+0.05) 0.37(0.29+0.06) +5.7% 2000.7: git add -A (full-index-v4) 0.31(0.26+0.06) 0.33(0.27+0.06) +6.5% 2000.8: git add -A (sparse-index-v3) 0.57(0.53+0.07) 0.05(0.04+0.08) -91.2% 2000.9: git add -A (sparse-index-v4) 0.58(0.55+0.06) 0.05(0.05+0.06) -91.4% While the 91% improvement seems impressive, it's important to recognize that previously we had significant overhead for expanding the sparse-index. Comparing to the full index case, 'git add -A' goes from 0.37s to 0.05s, which is "only" an 86% improvement. This modification to 'git add' creates some behavior change depending on the use of a sparse index. We modify a test in t1092 to demonstrate these changes which will be remedied in future changes. Reviewed-by: Elijah Newren Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- builtin/add.c | 3 +++ t/t1092-sparse-checkout-compatibility.sh | 25 +++++++++++++++++------- 2 files changed, 21 insertions(+), 7 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index b773b5a499..c76e6ddd35 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -528,6 +528,9 @@ int cmd_add(int argc, const char **argv, const char *prefix) add_new_files = !take_worktree_changes && !refresh_only && !add_renormalize; require_pathspec = !(take_worktree_changes || (0 < addremove_explicit)); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + hold_locked_index(&lock_file, LOCK_DIE_ON_ERROR); /* diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 4c3bcb3499..77343cb6d9 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -340,21 +340,27 @@ test_expect_success 'status/add: outside sparse cone' ' test_sparse_match git status --porcelain=v2 && - # This "git add folder1/a" fails with a warning - # in the sparse repos, differing from the full - # repo. This is intentional. + # Adding the path outside of the sparse-checkout cone should fail. test_sparse_match test_must_fail git add folder1/a && - test_sparse_match test_must_fail git add --refresh folder1/a && - test_all_match git status --porcelain=v2 && + + test_must_fail git -C sparse-checkout add --refresh folder1/a 2>sparse-checkout-err && + test_must_fail git -C sparse-index add --refresh folder1/a 2>sparse-index-err && + # NEEDSWORK: A sparse index changes the error message. + ! test_cmp sparse-checkout-err sparse-index-err && + + # NEEDSWORK: Adding a newly-tracked file outside the cone succeeds + test_sparse_match git add folder1/new && test_all_match git add . && test_all_match git status --porcelain=v2 && test_all_match git commit -m folder1/new && + test_all_match git rev-parse HEAD^{tree} && run_on_all ../edit-contents folder1/newer && test_all_match git add folder1/ && test_all_match git status --porcelain=v2 && - test_all_match git commit -m folder1/newer + test_all_match git commit -m folder1/newer && + test_all_match git rev-parse HEAD^{tree} ' test_expect_success 'checkout and reset --hard' ' @@ -641,7 +647,12 @@ test_expect_success 'sparse-index is not expanded' ' git -C sparse-index reset --hard && ensure_not_expanded checkout rename-out-to-out -- deep/deeper1 && git -C sparse-index reset --hard && - ensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 + ensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 && + + echo >>sparse-index/README.md && + ensure_not_expanded add -A && + echo >>sparse-index/extra.txt && + ensure_not_expanded add extra.txt ' # NEEDSWORK: a sparse-checkout behaves differently from a full checkout From 4eaffd81a5fbffc692f1044374a3a16689fc37a5 Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 29 Jul 2021 14:52:05 +0000 Subject: [PATCH 037/762] pathspec: stop calling ensure_full_index The add_pathspec_matches_against_index() focuses on matching a pathspec to file entries in the index. This already works correctly for its only use: checking if untracked files exist in the index. The compatibility checks in t1092 already test that 'git add ' works for a directory outside of the sparse cone. That provides coverage for removing this guard. This finalizes our ability to run 'git add .' without expanding a sparse index to a full one. This is evidenced by an update to t1092 and by these performance numbers for p2000-sparse-operations.sh: Test HEAD~1 HEAD -------------------------------------------------------------------------------- 2000.10: git add . (full-index-v3) 0.37(0.28+0.07) 0.36(0.27+0.06) -2.7% 2000.11: git add . (full-index-v4) 0.33(0.26+0.06) 0.32(0.28+0.05) -3.0% 2000.12: git add . (sparse-index-v3) 0.57(0.53+0.07) 0.06(0.06+0.07) -89.5% 2000.13: git add . (sparse-index-v4) 0.57(0.53+0.07) 0.05(0.03+0.09) -91.2% While the ~90% improvement is shown by the test results, it is worth noting that expanding the sparse index was adding overhead in previous commits. Comparing to the full index case, we see the performance go from 0.33s to 0.05s, an 85% improvement. Reviewed-by: Elijah Newren Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- pathspec.c | 2 -- t/t1092-sparse-checkout-compatibility.sh | 7 +++---- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/pathspec.c b/pathspec.c index 08f8d3eedc..44306fdaca 100644 --- a/pathspec.c +++ b/pathspec.c @@ -37,8 +37,6 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec, num_unmatched++; if (!num_unmatched) return; - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); for (i = 0; i < istate->cache_nr; i++) { const struct cache_entry *ce = istate->cache[i]; if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce)) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 77343cb6d9..dee20db5bb 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -322,9 +322,6 @@ test_expect_success 'commit including unstaged changes' ' test_expect_success 'status/add: outside sparse cone' ' init_repos && - # adding a "missing" file outside the cone should fail - test_sparse_match test_must_fail git add folder1/a && - # folder1 is at HEAD, but outside the sparse cone run_on_sparse mkdir folder1 && cp initial-repo/folder1/a sparse-checkout/folder1/a && @@ -652,7 +649,9 @@ test_expect_success 'sparse-index is not expanded' ' echo >>sparse-index/README.md && ensure_not_expanded add -A && echo >>sparse-index/extra.txt && - ensure_not_expanded add extra.txt + ensure_not_expanded add extra.txt && + echo >>sparse-index/untracked.txt && + ensure_not_expanded add . ' # NEEDSWORK: a sparse-checkout behaves differently from a full checkout From 939fa07582a2c9455e71f599263e2dcbe1d655b5 Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 29 Jul 2021 14:52:06 +0000 Subject: [PATCH 038/762] add: ignore outside the sparse-checkout in refresh() Since b243012 (refresh_index(): add flag to ignore SKIP_WORKTREE entries, 2021-04-08), 'git add --refresh ' will output a warning message when the path is outside the sparse-checkout definition. The implementation of this warning happened in parallel with the sparse-index work to add ensure_full_index() calls throughout the codebase. Update this loop to have the proper logic that checks to see if the pathspec is outside the sparse-checkout definition. This avoids the need to expand the sparse directory entry and determine if the path is tracked, untracked, or ignored. We simply avoid updating the stat() information because there isn't even an entry that matches the path! Reviewed-by: Elijah Newren Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- builtin/add.c | 10 +++++++++- t/t1092-sparse-checkout-compatibility.sh | 6 +----- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index c76e6ddd35..d512ece655 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -192,13 +192,21 @@ static int refresh(int verbose, const struct pathspec *pathspec) struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP; int flags = REFRESH_IGNORE_SKIP_WORKTREE | (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET); + struct pattern_list pl = { 0 }; + int sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl); seen = xcalloc(pathspec->nr, 1); refresh_index(&the_index, flags, pathspec, seen, _("Unstaged changes after refreshing the index:")); for (i = 0; i < pathspec->nr; i++) { if (!seen[i]) { - if (matches_skip_worktree(pathspec, i, &skip_worktree_seen)) { + const char *path = pathspec->items[i].original; + int dtype = DT_REG; + + if (matches_skip_worktree(pathspec, i, &skip_worktree_seen) || + (sparse_checkout_enabled && + !path_matches_pattern_list(path, strlen(path), NULL, + &dtype, &pl, &the_index))) { string_list_append(&only_match_skip_worktree, pathspec->items[i].original); } else { diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index dee20db5bb..ddc86bb415 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -339,11 +339,7 @@ test_expect_success 'status/add: outside sparse cone' ' # Adding the path outside of the sparse-checkout cone should fail. test_sparse_match test_must_fail git add folder1/a && - - test_must_fail git -C sparse-checkout add --refresh folder1/a 2>sparse-checkout-err && - test_must_fail git -C sparse-index add --refresh folder1/a 2>sparse-index-err && - # NEEDSWORK: A sparse index changes the error message. - ! test_cmp sparse-checkout-err sparse-index-err && + test_sparse_match test_must_fail git add --refresh folder1/a && # NEEDSWORK: Adding a newly-tracked file outside the cone succeeds test_sparse_match git add folder1/new && From 42f8ed6ca24854141c34aa82472ca126fdaeaf65 Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 29 Jul 2021 14:52:07 +0000 Subject: [PATCH 039/762] add: remove ensure_full_index() with --renormalize The --renormalize option updates the EOL conversions for the tracked files. However, the loop already ignores files marked with the SKIP_WORKTREE bit, so it will continue to do so with a sparse index because the sparse directory entries also have this bit set. Reviewed-by: Elijah Newren Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- builtin/add.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index d512ece655..c49e179abc 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -144,8 +144,6 @@ static int renormalize_tracked_files(const struct pathspec *pathspec, int flags) { int i, retval = 0; - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(&the_index); for (i = 0; i < active_nr; i++) { struct cache_entry *ce = active_cache[i]; From 7afc0b03a2307aa02b7caff2746664e886f38d9a Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Fri, 30 Jul 2021 11:47:36 +0000 Subject: [PATCH 040/762] merge-ort: rename str{map,intmap,set}_func() In order to make it clearer that these three variables holding a function refer to functions that will clear the strmap/strintmap/strset, rename them to str{map,intmap,set}_clear_func(). Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- merge-ort.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/merge-ort.c b/merge-ort.c index e75b524153..401a40247a 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -519,11 +519,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, { struct rename_info *renames = &opti->renames; int i; - void (*strmap_func)(struct strmap *, int) = + void (*strmap_clear_func)(struct strmap *, int) = reinitialize ? strmap_partial_clear : strmap_clear; - void (*strintmap_func)(struct strintmap *) = + void (*strintmap_clear_func)(struct strintmap *) = reinitialize ? strintmap_partial_clear : strintmap_clear; - void (*strset_func)(struct strset *) = + void (*strset_clear_func)(struct strset *) = reinitialize ? strset_partial_clear : strset_clear; /* @@ -534,14 +534,14 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, * to deallocate them. */ free_strmap_strings(&opti->paths); - strmap_func(&opti->paths, 1); + strmap_clear_func(&opti->paths, 1); /* * All keys and values in opti->conflicted are a subset of those in * opti->paths. We don't want to deallocate anything twice, so we * don't free the keys and we pass 0 for free_values. */ - strmap_func(&opti->conflicted, 0); + strmap_clear_func(&opti->conflicted, 0); /* * opti->paths_to_free is similar to opti->paths; we created it with @@ -559,24 +559,24 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, /* Free memory used by various renames maps */ for (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) { - strintmap_func(&renames->dirs_removed[i]); - strmap_func(&renames->dir_renames[i], 0); - strintmap_func(&renames->relevant_sources[i]); + strintmap_clear_func(&renames->dirs_removed[i]); + strmap_clear_func(&renames->dir_renames[i], 0); + strintmap_clear_func(&renames->relevant_sources[i]); if (!reinitialize) assert(renames->cached_pairs_valid_side == 0); if (i != renames->cached_pairs_valid_side && -1 != renames->cached_pairs_valid_side) { - strset_func(&renames->cached_target_names[i]); - strmap_func(&renames->cached_pairs[i], 1); - strset_func(&renames->cached_irrelevant[i]); + strset_clear_func(&renames->cached_target_names[i]); + strmap_clear_func(&renames->cached_pairs[i], 1); + strset_clear_func(&renames->cached_irrelevant[i]); partial_clear_dir_rename_count(&renames->dir_rename_count[i]); if (!reinitialize) strmap_clear(&renames->dir_rename_count[i], 1); } } for (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) { - strintmap_func(&renames->deferred[i].possible_trivial_merges); - strset_func(&renames->deferred[i].target_dirs); + strintmap_clear_func(&renames->deferred[i].possible_trivial_merges); + strset_clear_func(&renames->deferred[i].target_dirs); renames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */ } renames->cached_pairs_valid_side = 0; From fa0e936fbb683dbce70b797ae8a5be6b88f48d88 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Fri, 30 Jul 2021 11:47:37 +0000 Subject: [PATCH 041/762] diffcore-rename: use a mem_pool for exact rename detection's hashmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Exact rename detection, via insert_file_table(), uses a hashmap to store files by oid. Use a mem_pool for the hashmap entries so these can all be allocated and deallocated together. For the testcases mentioned in commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2020-10-28), this change improves the performance as follows: Before After no-renames: 204.2 ms ± 3.0 ms 202.5 ms ± 3.2 ms mega-renames: 1.076 s ± 0.015 s 1.072 s ± 0.012 s just-one-mega: 364.1 ms ± 7.0 ms 357.3 ms ± 3.9 ms Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- diffcore-rename.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/diffcore-rename.c b/diffcore-rename.c index 4ef0459cfb..73d884099e 100644 --- a/diffcore-rename.c +++ b/diffcore-rename.c @@ -317,10 +317,11 @@ static int find_identical_files(struct hashmap *srcs, } static void insert_file_table(struct repository *r, + struct mem_pool *pool, struct hashmap *table, int index, struct diff_filespec *filespec) { - struct file_similarity *entry = xmalloc(sizeof(*entry)); + struct file_similarity *entry = mem_pool_alloc(pool, sizeof(*entry)); entry->index = index; entry->filespec = filespec; @@ -336,7 +337,8 @@ static void insert_file_table(struct repository *r, * and then during the second round we try to match * cache-dirty entries as well. */ -static int find_exact_renames(struct diff_options *options) +static int find_exact_renames(struct diff_options *options, + struct mem_pool *pool) { int i, renames = 0; struct hashmap file_table; @@ -346,7 +348,7 @@ static int find_exact_renames(struct diff_options *options) */ hashmap_init(&file_table, NULL, NULL, rename_src_nr); for (i = rename_src_nr-1; i >= 0; i--) - insert_file_table(options->repo, + insert_file_table(options->repo, pool, &file_table, i, rename_src[i].p->one); @@ -354,8 +356,8 @@ static int find_exact_renames(struct diff_options *options) for (i = 0; i < rename_dst_nr; i++) renames += find_identical_files(&file_table, i, options); - /* Free the hash data structure and entries */ - hashmap_clear_and_free(&file_table, struct file_similarity, entry); + /* Free the hash data structure (entries will be freed with the pool) */ + hashmap_clear(&file_table); return renames; } @@ -1341,6 +1343,7 @@ void diffcore_rename_extended(struct diff_options *options, int num_destinations, dst_cnt; int num_sources, want_copies; struct progress *progress = NULL; + struct mem_pool local_pool; struct dir_rename_info info; struct diff_populate_filespec_options dpf_options = { .check_binary = 0, @@ -1409,11 +1412,18 @@ void diffcore_rename_extended(struct diff_options *options, goto cleanup; /* nothing to do */ trace2_region_enter("diff", "exact renames", options->repo); + mem_pool_init(&local_pool, 32*1024); /* * We really want to cull the candidates list early * with cheap tests in order to avoid doing deltas. */ - rename_count = find_exact_renames(options); + rename_count = find_exact_renames(options, &local_pool); + /* + * Discard local_pool immediately instead of at "cleanup:" in order + * to reduce maximum memory usage; inexact rename detection uses up + * a fair amount of memory, and mem_pools can too. + */ + mem_pool_discard(&local_pool, 0); trace2_region_leave("diff", "exact renames", options->repo); /* Did we only want exact renames? */ From cdf2241c717b05845ff9e25b87be6ca7559e82eb Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Fri, 30 Jul 2021 11:47:38 +0000 Subject: [PATCH 042/762] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers Make the code more flexible so that it can handle both being run with or without a memory pool by adding utility functions which will either call xmalloc, xcalloc, xstrndup or mem_pool_alloc, mem_pool_calloc, mem_pool_strndup depending on whether we have a non-NULL memory pool. A subsequent commit will make use of these. (We will actually be dropping these functions soon and just assuming we always have a memory pool, but the flexibility was very useful during development of merge-ort so I want to be able to restore it if needed.) Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- merge-ort.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/merge-ort.c b/merge-ort.c index 401a40247a..63f67246d3 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -664,6 +664,30 @@ static void path_msg(struct merge_options *opt, strbuf_addch(sb, '\n'); } +MAYBE_UNUSED +static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size) +{ + if (!pool) + return xcalloc(count, size); + return mem_pool_calloc(pool, count, size); +} + +MAYBE_UNUSED +static void *pool_alloc(struct mem_pool *pool, size_t size) +{ + if (!pool) + return xmalloc(size); + return mem_pool_alloc(pool, size); +} + +MAYBE_UNUSED +static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len) +{ + if (!pool) + return xstrndup(str, len); + return mem_pool_strndup(pool, str, len); +} + /* add a string to a strbuf, but converting "/" to "_" */ static void add_flattened_path(struct strbuf *out, const char *s) { From 4137c54b902faa640c485dc6d20eaac946e503a5 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Fri, 30 Jul 2021 11:47:39 +0000 Subject: [PATCH 043/762] merge-ort: set up a memory pool merge-ort has a lot of data structures, and they all tend to be freed together in clear_or_reinit_internal_opts(). Set up a memory pool to allow us to make these allocations and deallocations faster. Future commits will adjust various callers to make use of this memory pool. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- merge-ort.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/merge-ort.c b/merge-ort.c index 63f67246d3..3f42543626 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -37,6 +37,8 @@ #include "unpack-trees.h" #include "xdiff-interface.h" +#define USE_MEMORY_POOL 1 /* faster, but obscures memory leak hunting */ + /* * We have many arrays of size 3. Whenever we have such an array, the * indices refer to one of the sides of the three-way merge. This is so @@ -339,6 +341,17 @@ struct merge_options_internal { */ struct strmap conflicted; + /* + * pool: memory pool for fast allocation/deallocation + * + * We allocate room for lots of filenames and auxiliary data + * structures in merge_options_internal, and it tends to all be + * freed together too. Using a memory pool for these provides a + * nice speedup. + */ + struct mem_pool internal_pool; + struct mem_pool *pool; /* NULL, or pointer to internal_pool */ + /* * paths_to_free: additional list of strings to free * @@ -603,6 +616,12 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, strmap_clear(&opti->output, 0); } +#if USE_MEMORY_POOL + mem_pool_discard(&opti->internal_pool, 0); + if (!reinitialize) + opti->pool = NULL; +#endif + /* Clean out callback_data as well. */ FREE_AND_NULL(renames->callback_data); renames->callback_data_nr = renames->callback_data_alloc = 0; @@ -4381,6 +4400,12 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) /* Initialization of various renames fields */ renames = &opt->priv->renames; +#if USE_MEMORY_POOL + mem_pool_init(&opt->priv->internal_pool, 0); + opt->priv->pool = &opt->priv->internal_pool; +#else + opt->priv->pool = NULL; +#endif for (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) { strintmap_init_with_options(&renames->dirs_removed[i], NOT_RELEVANT, NULL, 0); From 6697ee01b5d31df5c83ace9eabb6285cf42ff172 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Fri, 30 Jul 2021 11:47:40 +0000 Subject: [PATCH 044/762] merge-ort: switch our strmaps over to using memory pools MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For all the strmaps (including strintmaps and strsets) whose memory is unconditionally freed as part of clear_or_reinit_internal_opts(), switch them over to using our new memory pool. For the testcases mentioned in commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2020-10-28), this change improves the performance as follows: Before After no-renames: 202.5 ms ± 3.2 ms 198.1 ms ± 2.6 ms mega-renames: 1.072 s ± 0.012 s 715.8 ms ± 4.0 ms just-one-mega: 357.3 ms ± 3.9 ms 276.8 ms ± 4.2 ms Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- merge-ort.c | 125 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 75 insertions(+), 50 deletions(-) diff --git a/merge-ort.c b/merge-ort.c index 3f42543626..99c7569085 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -539,15 +539,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, void (*strset_clear_func)(struct strset *) = reinitialize ? strset_partial_clear : strset_clear; - /* - * We marked opti->paths with strdup_strings = 0, so that we - * wouldn't have to make another copy of the fullpath created by - * make_traverse_path from setup_path_info(). But, now that we've - * used it and have no other references to these strings, it is time - * to deallocate them. - */ - free_strmap_strings(&opti->paths); - strmap_clear_func(&opti->paths, 1); + if (opti->pool) + strmap_clear_func(&opti->paths, 0); + else { + /* + * We marked opti->paths with strdup_strings = 0, so that + * we wouldn't have to make another copy of the fullpath + * created by make_traverse_path from setup_path_info(). + * But, now that we've used it and have no other references + * to these strings, it is time to deallocate them. + */ + free_strmap_strings(&opti->paths); + strmap_clear_func(&opti->paths, 1); + } /* * All keys and values in opti->conflicted are a subset of those in @@ -556,16 +560,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, */ strmap_clear_func(&opti->conflicted, 0); - /* - * opti->paths_to_free is similar to opti->paths; we created it with - * strdup_strings = 0 to avoid making _another_ copy of the fullpath - * but now that we've used it and have no other references to these - * strings, it is time to deallocate them. We do so by temporarily - * setting strdup_strings to 1. - */ - opti->paths_to_free.strdup_strings = 1; - string_list_clear(&opti->paths_to_free, 0); - opti->paths_to_free.strdup_strings = 0; + if (!opti->pool) { + /* + * opti->paths_to_free is similar to opti->paths; we + * created it with strdup_strings = 0 to avoid making + * _another_ copy of the fullpath but now that we've used + * it and have no other references to these strings, it is + * time to deallocate them. We do so by temporarily + * setting strdup_strings to 1. + */ + opti->paths_to_free.strdup_strings = 1; + string_list_clear(&opti->paths_to_free, 0); + opti->paths_to_free.strdup_strings = 0; + } if (opti->attr_index.cache_nr) /* true iff opt->renormalize */ discard_index(&opti->attr_index); @@ -683,7 +690,6 @@ static void path_msg(struct merge_options *opt, strbuf_addch(sb, '\n'); } -MAYBE_UNUSED static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size) { if (!pool) @@ -691,7 +697,6 @@ static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size) return mem_pool_calloc(pool, count, size); } -MAYBE_UNUSED static void *pool_alloc(struct mem_pool *pool, size_t size) { if (!pool) @@ -699,7 +704,6 @@ static void *pool_alloc(struct mem_pool *pool, size_t size) return mem_pool_alloc(pool, size); } -MAYBE_UNUSED static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len) { if (!pool) @@ -835,8 +839,9 @@ static void setup_path_info(struct merge_options *opt, assert(!df_conflict || !resolved); /* df_conflict implies !resolved */ assert(resolved == (merged_version != NULL)); - mi = xcalloc(1, resolved ? sizeof(struct merged_info) : - sizeof(struct conflict_info)); + mi = pool_calloc(opt->priv->pool, 1, + resolved ? sizeof(struct merged_info) : + sizeof(struct conflict_info)); mi->directory_name = current_dir_name; mi->basename_offset = current_dir_name_len; mi->clean = !!resolved; @@ -1128,7 +1133,7 @@ static int collect_merge_info_callback(int n, len = traverse_path_len(info, p->pathlen); /* +1 in both of the following lines to include the NUL byte */ - fullpath = xmalloc(len + 1); + fullpath = pool_alloc(opt->priv->pool, len + 1); make_traverse_path(fullpath, len + 1, info, p->path, p->pathlen); /* @@ -1383,7 +1388,7 @@ static int handle_deferred_entries(struct merge_options *opt, copy = renames->deferred[side].possible_trivial_merges; strintmap_init_with_options(&renames->deferred[side].possible_trivial_merges, 0, - NULL, + opt->priv->pool, 0); strintmap_for_each_entry(©, &iter, entry) { const char *path = entry->key; @@ -2335,12 +2340,21 @@ static void apply_directory_rename_modifications(struct merge_options *opt, VERIFY_CI(ci); /* Find parent directories missing from opt->priv->paths */ - cur_path = new_path; + if (opt->priv->pool) { + cur_path = mem_pool_strdup(opt->priv->pool, new_path); + free((char*)new_path); + new_path = (char *)cur_path; + } else { + cur_path = new_path; + } + while (1) { /* Find the parent directory of cur_path */ char *last_slash = strrchr(cur_path, '/'); if (last_slash) { - parent_name = xstrndup(cur_path, last_slash - cur_path); + parent_name = pool_strndup(opt->priv->pool, + cur_path, + last_slash - cur_path); } else { parent_name = opt->priv->toplevel_dir; break; @@ -2349,7 +2363,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt, /* Look it up in opt->priv->paths */ entry = strmap_get_entry(&opt->priv->paths, parent_name); if (entry) { - free((char*)parent_name); + if (!opt->priv->pool) + free((char*)parent_name); parent_name = entry->key; /* reuse known pointer */ break; } @@ -2376,12 +2391,15 @@ static void apply_directory_rename_modifications(struct merge_options *opt, parent_name = cur_dir; } - /* - * We are removing old_path from opt->priv->paths. old_path also will - * eventually need to be freed, but it may still be used by e.g. - * ci->pathnames. So, store it in another string-list for now. - */ - string_list_append(&opt->priv->paths_to_free, old_path); + if (!opt->priv->pool) { + /* + * We are removing old_path from opt->priv->paths. + * old_path also will eventually need to be freed, but it + * may still be used by e.g. ci->pathnames. So, store it + * in another string-list for now. + */ + string_list_append(&opt->priv->paths_to_free, old_path); + } assert(ci->filemask == 2 || ci->filemask == 4); assert(ci->dirmask == 0); @@ -2416,7 +2434,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt, new_ci->stages[index].mode = ci->stages[index].mode; oidcpy(&new_ci->stages[index].oid, &ci->stages[index].oid); - free(ci); + if (!opt->priv->pool) + free(ci); ci = new_ci; } @@ -3623,7 +3642,8 @@ static void process_entry(struct merge_options *opt, * the directory to remain here, so we need to move this * path to some new location. */ - CALLOC_ARRAY(new_ci, 1); + new_ci = pool_calloc(opt->priv->pool, 1, sizeof(*new_ci)); + /* We don't really want new_ci->merged.result copied, but it'll * be overwritten below so it doesn't matter. We also don't * want any directory mode/oid values copied, but we'll zero @@ -3715,7 +3735,7 @@ static void process_entry(struct merge_options *opt, const char *a_path = NULL, *b_path = NULL; int rename_a = 0, rename_b = 0; - new_ci = xmalloc(sizeof(*new_ci)); + new_ci = pool_alloc(opt->priv->pool, sizeof(*new_ci)); if (S_ISREG(a_mode)) rename_a = 1; @@ -3788,12 +3808,14 @@ static void process_entry(struct merge_options *opt, strmap_remove(&opt->priv->paths, path, 0); /* * We removed path from opt->priv->paths. path - * will also eventually need to be freed, but - * it may still be used by e.g. ci->pathnames. - * So, store it in another string-list for now. + * will also eventually need to be freed if not + * part of a memory pool...but it may still be + * used by e.g. ci->pathnames. So, store it in + * another string-list for now in that case. */ - string_list_append(&opt->priv->paths_to_free, - path); + if (!opt->priv->pool) + string_list_append(&opt->priv->paths_to_free, + path); } /* @@ -4335,6 +4357,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) { struct rename_info *renames; int i; + struct mem_pool *pool = NULL; /* Sanity checks on opt */ trace2_region_enter("merge", "sanity checks", opt->repo); @@ -4406,9 +4429,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) #else opt->priv->pool = NULL; #endif + pool = opt->priv->pool; for (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) { strintmap_init_with_options(&renames->dirs_removed[i], - NOT_RELEVANT, NULL, 0); + NOT_RELEVANT, pool, 0); strmap_init_with_options(&renames->dir_rename_count[i], NULL, 1); strmap_init_with_options(&renames->dir_renames[i], @@ -4422,7 +4446,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) */ strintmap_init_with_options(&renames->relevant_sources[i], -1 /* explicitly invalid */, - NULL, 0); + pool, 0); strmap_init_with_options(&renames->cached_pairs[i], NULL, 1); strset_init_with_options(&renames->cached_irrelevant[i], @@ -4432,9 +4456,9 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) } for (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) { strintmap_init_with_options(&renames->deferred[i].possible_trivial_merges, - 0, NULL, 0); + 0, pool, 0); strset_init_with_options(&renames->deferred[i].target_dirs, - NULL, 1); + pool, 1); renames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */ } @@ -4447,9 +4471,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) * In contrast, conflicted just has a subset of keys from paths, so * we don't want to free those (it'd be a duplicate free). */ - strmap_init_with_options(&opt->priv->paths, NULL, 0); - strmap_init_with_options(&opt->priv->conflicted, NULL, 0); - string_list_init_nodup(&opt->priv->paths_to_free); + strmap_init_with_options(&opt->priv->paths, pool, 0); + strmap_init_with_options(&opt->priv->conflicted, pool, 0); + if (!opt->priv->pool) + string_list_init_nodup(&opt->priv->paths_to_free); /* * keys & strbufs in output will sometimes need to outlive "paths", From a8791ef6492bf702ba70352009cbd28fb581c6e2 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Fri, 30 Jul 2021 11:47:41 +0000 Subject: [PATCH 045/762] diffcore-rename, merge-ort: add wrapper functions for filepair alloc/dealloc We want to be able to allocate filespecs and filepairs using a mem_pool. However, filespec data will still remain outside the pool (perhaps in the future we could plumb the pool through the various diff APIs to allocate the filespec data too, but for now we are limiting the scope). Add some extra functions to allocate these appropriately based on the non-NULL-ness of opt->priv->pool, as well as some extra functions to handle correctly deallocating the relevant parts of them. A future commit will make use of these new functions. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- diffcore-rename.c | 41 +++++++++++++++++++++++++++++++++++++++++ diffcore.h | 2 ++ merge-ort.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+) diff --git a/diffcore-rename.c b/diffcore-rename.c index 73d884099e..5bc559f79e 100644 --- a/diffcore-rename.c +++ b/diffcore-rename.c @@ -1328,6 +1328,47 @@ static void handle_early_known_dir_renames(struct dir_rename_info *info, rename_src_nr = new_num_src; } +static void free_filespec_data(struct diff_filespec *spec) +{ + if (!--spec->count) + diff_free_filespec_data(spec); +} + +MAYBE_UNUSED +static void pool_free_filespec(struct mem_pool *pool, + struct diff_filespec *spec) +{ + if (!pool) { + free_filespec(spec); + return; + } + + /* + * Similar to free_filespec(), but only frees the data. The spec + * itself was allocated in the pool and should not be individually + * freed. + */ + free_filespec_data(spec); +} + +MAYBE_UNUSED +void pool_diff_free_filepair(struct mem_pool *pool, + struct diff_filepair *p) +{ + if (!pool) { + diff_free_filepair(p); + return; + } + + /* + * Similar to diff_free_filepair() but only frees the data from the + * filespecs; not the filespecs or the filepair which were + * allocated from the pool. + */ + free_filespec_data(p->one); + free_filespec_data(p->two); +} + void diffcore_rename_extended(struct diff_options *options, struct strintmap *relevant_sources, struct strintmap *dirs_removed, diff --git a/diffcore.h b/diffcore.h index 533b30e21e..b58ee6b193 100644 --- a/diffcore.h +++ b/diffcore.h @@ -127,6 +127,8 @@ struct diff_filepair { #define DIFF_PAIR_MODE_CHANGED(p) ((p)->one->mode != (p)->two->mode) void diff_free_filepair(struct diff_filepair *); +void pool_diff_free_filepair(struct mem_pool *pool, + struct diff_filepair *p); int diff_unmodified_pair(struct diff_filepair *); diff --git a/merge-ort.c b/merge-ort.c index 99c7569085..e79830f918 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -690,6 +690,48 @@ static void path_msg(struct merge_options *opt, strbuf_addch(sb, '\n'); } +MAYBE_UNUSED +static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool, + const char *path) +{ + struct diff_filespec *spec; + size_t len; + + if (!pool) + return alloc_filespec(path); + + /* Same code as alloc_filespec, except allocate from pool */ + len = strlen(path); + + spec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1)); + memcpy(spec+1, path, len); + spec->path = (void*)(spec+1); + + spec->count = 1; + spec->is_binary = -1; + return spec; +} + +MAYBE_UNUSED +static struct diff_filepair *pool_diff_queue(struct mem_pool *pool, + struct diff_queue_struct *queue, + struct diff_filespec *one, + struct diff_filespec *two) +{ + struct diff_filepair *dp; + + if (!pool) + return diff_queue(queue, one, two); + + /* Same code as diff_queue, except allocate from pool */ + dp = mem_pool_calloc(pool, 1, sizeof(*dp)); + dp->one = one; + dp->two = two; + if (queue) + diff_q(queue, dp); + return dp; +} + static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size) { if (!pool) From f239fff4c183b231134115bc3b38b4f8e3982d3e Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Fri, 30 Jul 2021 11:47:42 +0000 Subject: [PATCH 046/762] merge-ort: store filepairs and filespecs in our mem_pool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For the testcases mentioned in commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2020-10-28), this change improves the performance as follows: Before After no-renames: 198.1 ms ± 2.6 ms 198.5 ms ± 3.4 ms mega-renames: 715.8 ms ± 4.0 ms 679.1 ms ± 5.6 ms just-one-mega: 276.8 ms ± 4.2 ms 271.9 ms ± 2.8 ms Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- diffcore-rename.c | 9 ++++----- diffcore.h | 1 + merge-ort.c | 26 ++++++++++++++------------ 3 files changed, 19 insertions(+), 17 deletions(-) diff --git a/diffcore-rename.c b/diffcore-rename.c index 5bc559f79e..7e6b3e1b14 100644 --- a/diffcore-rename.c +++ b/diffcore-rename.c @@ -1334,7 +1334,6 @@ static void free_filespec_data(struct diff_filespec *spec) diff_free_filespec_data(spec); } -MAYBE_UNUSED static void pool_free_filespec(struct mem_pool *pool, struct diff_filespec *spec) { @@ -1351,7 +1350,6 @@ static void pool_free_filespec(struct mem_pool *pool, free_filespec_data(spec); } -MAYBE_UNUSED void pool_diff_free_filepair(struct mem_pool *pool, struct diff_filepair *p) { @@ -1370,6 +1368,7 @@ void pool_diff_free_filepair(struct mem_pool *pool, } void diffcore_rename_extended(struct diff_options *options, + struct mem_pool *pool, struct strintmap *relevant_sources, struct strintmap *dirs_removed, struct strmap *dir_rename_count, @@ -1683,7 +1682,7 @@ void diffcore_rename_extended(struct diff_options *options, pair_to_free = p; if (pair_to_free) - diff_free_filepair(pair_to_free); + pool_diff_free_filepair(pool, pair_to_free); } diff_debug_queue("done copying original", &outq); @@ -1693,7 +1692,7 @@ void diffcore_rename_extended(struct diff_options *options, for (i = 0; i < rename_dst_nr; i++) if (rename_dst[i].filespec_to_free) - free_filespec(rename_dst[i].filespec_to_free); + pool_free_filespec(pool, rename_dst[i].filespec_to_free); cleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL); FREE_AND_NULL(rename_dst); @@ -1710,5 +1709,5 @@ void diffcore_rename_extended(struct diff_options *options, void diffcore_rename(struct diff_options *options) { - diffcore_rename_extended(options, NULL, NULL, NULL, NULL); + diffcore_rename_extended(options, NULL, NULL, NULL, NULL, NULL); } diff --git a/diffcore.h b/diffcore.h index b58ee6b193..badc2261c2 100644 --- a/diffcore.h +++ b/diffcore.h @@ -181,6 +181,7 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count); void diffcore_break(struct repository *, int); void diffcore_rename(struct diff_options *); void diffcore_rename_extended(struct diff_options *options, + struct mem_pool *pool, struct strintmap *relevant_sources, struct strintmap *dirs_removed, struct strmap *dir_rename_count, diff --git a/merge-ort.c b/merge-ort.c index e79830f918..f4f0a3d57f 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -690,7 +690,6 @@ static void path_msg(struct merge_options *opt, strbuf_addch(sb, '\n'); } -MAYBE_UNUSED static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool, const char *path) { @@ -712,7 +711,6 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool, return spec; } -MAYBE_UNUSED static struct diff_filepair *pool_diff_queue(struct mem_pool *pool, struct diff_queue_struct *queue, struct diff_filespec *one, @@ -930,6 +928,7 @@ static void add_pair(struct merge_options *opt, unsigned dir_rename_mask) { struct diff_filespec *one, *two; + struct mem_pool *pool = opt->priv->pool; struct rename_info *renames = &opt->priv->renames; int names_idx = is_add ? side : 0; @@ -980,11 +979,11 @@ static void add_pair(struct merge_options *opt, return; } - one = alloc_filespec(pathname); - two = alloc_filespec(pathname); + one = pool_alloc_filespec(pool, pathname); + two = pool_alloc_filespec(pool, pathname); fill_filespec(is_add ? two : one, &names[names_idx].oid, 1, names[names_idx].mode); - diff_queue(&renames->pairs[side], one, two); + pool_diff_queue(pool, &renames->pairs[side], one, two); } static void collect_rename_info(struct merge_options *opt, @@ -2893,6 +2892,7 @@ static void use_cached_pairs(struct merge_options *opt, { struct hashmap_iter iter; struct strmap_entry *entry; + struct mem_pool *pool = opt->priv->pool; /* * Add to side_pairs all entries from renames->cached_pairs[side_index]. @@ -2906,9 +2906,9 @@ static void use_cached_pairs(struct merge_options *opt, new_name = old_name; /* We don't care about oid/mode, only filenames and status */ - one = alloc_filespec(old_name); - two = alloc_filespec(new_name); - diff_queue(pairs, one, two); + one = pool_alloc_filespec(pool, old_name); + two = pool_alloc_filespec(pool, new_name); + pool_diff_queue(pool, pairs, one, two); pairs->queue[pairs->nr-1]->status = entry->value ? 'R' : 'D'; } } @@ -3016,6 +3016,7 @@ static int detect_regular_renames(struct merge_options *opt, diff_queued_diff = renames->pairs[side_index]; trace2_region_enter("diff", "diffcore_rename", opt->repo); diffcore_rename_extended(&diff_opts, + opt->priv->pool, &renames->relevant_sources[side_index], &renames->dirs_removed[side_index], &renames->dir_rename_count[side_index], @@ -3066,7 +3067,7 @@ static int collect_renames(struct merge_options *opt, if (p->status != 'A' && p->status != 'R') { possibly_cache_new_pair(renames, p, side_index, NULL); - diff_free_filepair(p); + pool_diff_free_filepair(opt->priv->pool, p); continue; } @@ -3079,7 +3080,7 @@ static int collect_renames(struct merge_options *opt, possibly_cache_new_pair(renames, p, side_index, new_path); if (p->status != 'R' && !new_path) { - diff_free_filepair(p); + pool_diff_free_filepair(opt->priv->pool, p); continue; } @@ -3197,7 +3198,7 @@ cleanup: side_pairs = &renames->pairs[s]; for (i = 0; i < side_pairs->nr; ++i) { struct diff_filepair *p = side_pairs->queue[i]; - diff_free_filepair(p); + pool_diff_free_filepair(opt->priv->pool, p); } } @@ -3210,7 +3211,8 @@ simple_cleanup: if (combined.nr) { int i; for (i = 0; i < combined.nr; i++) - diff_free_filepair(combined.queue[i]); + pool_diff_free_filepair(opt->priv->pool, + combined.queue[i]); free(combined.queue); } From 092e5115d1312c0b30348404d3f5b2bb02909342 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Fri, 30 Jul 2021 11:47:43 +0000 Subject: [PATCH 047/762] merge-ort: reuse path strings in pool_alloc_filespec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pool_alloc_filespec() was written so that the code when pool != NULL mimicked the code from alloc_filespec(), which including allocating enough extra space for the path and then copying it. However, the path passed to pool_alloc_filespec() is always going to already be in the same memory pool, so we may as well reuse it instead of copying it. For the testcases mentioned in commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2020-10-28), this change improves the performance as follows: Before After no-renames: 198.5 ms ± 3.4 ms 198.3 ms ± 2.9 ms mega-renames: 679.1 ms ± 5.6 ms 661.8 ms ± 5.9 ms just-one-mega: 271.9 ms ± 2.8 ms 264.6 ms ± 2.5 ms Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- merge-ort.c | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/merge-ort.c b/merge-ort.c index f4f0a3d57f..86ab8f6012 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -694,17 +694,13 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool, const char *path) { struct diff_filespec *spec; - size_t len; if (!pool) return alloc_filespec(path); - /* Same code as alloc_filespec, except allocate from pool */ - len = strlen(path); - - spec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1)); - memcpy(spec+1, path, len); - spec->path = (void*)(spec+1); + /* Similar to alloc_filespec, but allocate from pool and reuse path */ + spec = mem_pool_calloc(pool, 1, sizeof(*spec)); + spec->path = (char*)path; /* spec won't modify it */ spec->count = 1; spec->is_binary = -1; @@ -2904,6 +2900,25 @@ static void use_cached_pairs(struct merge_options *opt, const char *new_name = entry->value; if (!new_name) new_name = old_name; + if (pool) { + /* + * cached_pairs has _copies* of old_name and new_name, + * because it has to persist across merges. When + * pool != NULL + * pool_alloc_filespec() will just re-use the existing + * filenames, which will also get re-used by + * opt->priv->paths if they become renames, and then + * get freed at the end of the merge, leaving the copy + * in cached_pairs dangling. Avoid this by making a + * copy here. + * + * When pool == NULL, pool_alloc_filespec() calls + * alloc_filespec(), which makes a copy; we don't want + * to add another. + */ + old_name = mem_pool_strdup(pool, old_name); + new_name = mem_pool_strdup(pool, new_name); + } /* We don't care about oid/mode, only filenames and status */ one = pool_alloc_filespec(pool, old_name); From 1119a15b5c8521e75c412a129cd6318285cac773 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 30 Jul 2021 11:31:53 +0200 Subject: [PATCH 048/762] http: drop support for curl < 7.11.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop support for this ancient version of curl and simplify the code by allowing us get rid of some "#ifdef"'s. Git will not build with vanilla curl older than 7.11.1 due our use of CURLOPT_POSTFIELDSIZE in 37ee680d9b (http.postbuffer: allow full range of ssize_t values, 2017-04-11). This field was introduced in curl 7.11.1. We could solve these compilation problems with more #ifdefs, but it's not worth the trouble. Version 7.11.1 came out in March of 2004, over 17 years ago. Let's declare that too old and drop any existing ifdefs that go further back. One obvious benefit is that we'll have fewer conditional bits cluttering the code. This patch drops all #ifdefs that reference older versions (note that curl's preprocessor macros are in hex, so we're looking for 070b01, not 071101). Signed-off-by: Jeff King Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- http.c | 53 --------------------------------------------------- http.h | 12 +----------- remote-curl.c | 3 --- 3 files changed, 1 insertion(+), 67 deletions(-) diff --git a/http.c b/http.c index 8119247149..56182a89e2 100644 --- a/http.c +++ b/http.c @@ -19,19 +19,11 @@ static struct trace_key trace_curl = TRACE_KEY_INIT(CURL); static int trace_curl_data = 1; static int trace_curl_redact = 1; -#if LIBCURL_VERSION_NUM >= 0x070a08 long int git_curl_ipresolve = CURL_IPRESOLVE_WHATEVER; -#else -long int git_curl_ipresolve; -#endif int active_requests; int http_is_verbose; ssize_t http_post_buffer = 16 * LARGE_PACKET_MAX; -#if LIBCURL_VERSION_NUM >= 0x070a06 -#define LIBCURL_CAN_HANDLE_AUTH_ANY -#endif - static int min_curl_sessions = 1; static int curl_session_count; #ifdef USE_CURL_MULTI @@ -68,15 +60,9 @@ static struct { { "tlsv1.3", CURL_SSLVERSION_TLSv1_3 }, #endif }; -#if LIBCURL_VERSION_NUM >= 0x070903 static const char *ssl_key; -#endif -#if LIBCURL_VERSION_NUM >= 0x070908 static const char *ssl_capath; -#endif -#if LIBCURL_VERSION_NUM >= 0x071304 static const char *curl_no_proxy; -#endif #if LIBCURL_VERSION_NUM >= 0x072c00 static const char *ssl_pinnedkey; #endif @@ -101,9 +87,7 @@ static struct { { "digest", CURLAUTH_DIGEST }, { "negotiate", CURLAUTH_GSSNEGOTIATE }, { "ntlm", CURLAUTH_NTLM }, -#ifdef LIBCURL_CAN_HANDLE_AUTH_ANY { "anyauth", CURLAUTH_ANY }, -#endif /* * CURLAUTH_DIGEST_IE has no corresponding command-line option in * curl(1) and is not included in CURLAUTH_ANY, so we leave it out @@ -143,7 +127,6 @@ enum http_follow_config http_follow_config = HTTP_FOLLOW_INITIAL; static struct credential cert_auth = CREDENTIAL_INIT; static int ssl_cert_password_required; -#ifdef LIBCURL_CAN_HANDLE_AUTH_ANY static unsigned long http_auth_methods = CURLAUTH_ANY; static int http_auth_methods_restricted; /* Modes for which empty_auth cannot actually help us. */ @@ -153,7 +136,6 @@ static unsigned long empty_auth_useless = | CURLAUTH_DIGEST_IE #endif | CURLAUTH_DIGEST; -#endif static struct curl_slist *pragma_header; static struct curl_slist *no_pragma_header; @@ -237,12 +219,8 @@ static void finish_active_slot(struct active_request_slot *slot) if (slot->results != NULL) { slot->results->curl_result = slot->curl_result; slot->results->http_code = slot->http_code; -#if LIBCURL_VERSION_NUM >= 0x070a08 curl_easy_getinfo(slot->curl, CURLINFO_HTTPAUTH_AVAIL, &slot->results->auth_avail); -#else - slot->results->auth_avail = 0; -#endif curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CONNECTCODE, &slot->results->http_connectcode); @@ -305,14 +283,10 @@ static int http_options(const char *var, const char *value, void *cb) return git_config_string(&ssl_version, var, value); if (!strcmp("http.sslcert", var)) return git_config_pathname(&ssl_cert, var, value); -#if LIBCURL_VERSION_NUM >= 0x070903 if (!strcmp("http.sslkey", var)) return git_config_pathname(&ssl_key, var, value); -#endif -#if LIBCURL_VERSION_NUM >= 0x070908 if (!strcmp("http.sslcapath", var)) return git_config_pathname(&ssl_capath, var, value); -#endif if (!strcmp("http.sslcainfo", var)) return git_config_pathname(&ssl_cainfo, var, value); if (!strcmp("http.sslcertpasswordprotected", var)) { @@ -461,12 +435,6 @@ static int curl_empty_auth_enabled(void) if (curl_empty_auth >= 0) return curl_empty_auth; -#ifndef LIBCURL_CAN_HANDLE_AUTH_ANY - /* - * Our libcurl is too old to do AUTH_ANY in the first place; - * just default to turning the feature off. - */ -#else /* * In the automatic case, kick in the empty-auth * hack as long as we would potentially try some @@ -479,7 +447,6 @@ static int curl_empty_auth_enabled(void) if (http_auth_methods_restricted && (http_auth_methods & ~empty_auth_useless)) return 1; -#endif return 0; } @@ -552,7 +519,6 @@ static void init_curl_proxy_auth(CURL *result) var_override(&http_proxy_authmethod, getenv("GIT_HTTP_PROXY_AUTHMETHOD")); -#if LIBCURL_VERSION_NUM >= 0x070a07 /* CURLOPT_PROXYAUTH and CURLAUTH_ANY */ if (http_proxy_authmethod) { int i; for (i = 0; i < ARRAY_SIZE(proxy_authmethods); i++) { @@ -570,7 +536,6 @@ static void init_curl_proxy_auth(CURL *result) } else curl_easy_setopt(result, CURLOPT_PROXYAUTH, CURLAUTH_ANY); -#endif } static int has_cert_password(void) @@ -879,12 +844,8 @@ static CURL *get_curl_handle(void) } #endif -#if LIBCURL_VERSION_NUM >= 0x070907 curl_easy_setopt(result, CURLOPT_NETRC, CURL_NETRC_OPTIONAL); -#endif -#ifdef LIBCURL_CAN_HANDLE_AUTH_ANY curl_easy_setopt(result, CURLOPT_HTTPAUTH, CURLAUTH_ANY); -#endif #ifdef CURLGSSAPI_DELEGATION_FLAG if (curl_deleg) { @@ -940,14 +901,10 @@ static CURL *get_curl_handle(void) curl_easy_setopt(result, CURLOPT_SSLCERT, ssl_cert); if (has_cert_password()) curl_easy_setopt(result, CURLOPT_KEYPASSWD, cert_auth.password); -#if LIBCURL_VERSION_NUM >= 0x070903 if (ssl_key != NULL) curl_easy_setopt(result, CURLOPT_SSLKEY, ssl_key); -#endif -#if LIBCURL_VERSION_NUM >= 0x070908 if (ssl_capath != NULL) curl_easy_setopt(result, CURLOPT_CAPATH, ssl_capath); -#endif #if LIBCURL_VERSION_NUM >= 0x072c00 if (ssl_pinnedkey != NULL) curl_easy_setopt(result, CURLOPT_PINNEDPUBLICKEY, ssl_pinnedkey); @@ -1180,12 +1137,8 @@ void http_init(struct remote *remote, const char *url, int proactive_auth) curl_ssl_verify = 0; set_from_env(&ssl_cert, "GIT_SSL_CERT"); -#if LIBCURL_VERSION_NUM >= 0x070903 set_from_env(&ssl_key, "GIT_SSL_KEY"); -#endif -#if LIBCURL_VERSION_NUM >= 0x070908 set_from_env(&ssl_capath, "GIT_SSL_CAPATH"); -#endif set_from_env(&ssl_cainfo, "GIT_SSL_CAINFO"); set_from_env(&user_agent, "GIT_HTTP_USER_AGENT"); @@ -1367,12 +1320,8 @@ struct active_request_slot *get_active_slot(void) else curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 0); -#if LIBCURL_VERSION_NUM >= 0x070a08 curl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve); -#endif -#ifdef LIBCURL_CAN_HANDLE_AUTH_ANY curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods); -#endif if (http_auth.password || curl_empty_auth_enabled()) init_curl_http_auth(slot->curl); @@ -1654,13 +1603,11 @@ static int handle_curl_result(struct slot_results *results) credential_reject(&http_auth); return HTTP_NOAUTH; } else { -#ifdef LIBCURL_CAN_HANDLE_AUTH_ANY http_auth_methods &= ~CURLAUTH_GSSNEGOTIATE; if (results->auth_avail) { http_auth_methods &= results->auth_avail; http_auth_methods_restricted = 1; } -#endif return HTTP_REAUTH; } } else { diff --git a/http.h b/http.h index bf3d1270ad..d2f8cc5661 100644 --- a/http.h +++ b/http.h @@ -22,13 +22,7 @@ #define DEFAULT_MAX_REQUESTS 5 #endif -#if LIBCURL_VERSION_NUM < 0x070704 -#define curl_global_cleanup() do { /* nothing */ } while (0) -#endif - -#if LIBCURL_VERSION_NUM < 0x070800 -#define curl_global_init(a) do { /* nothing */ } while (0) -#elif LIBCURL_VERSION_NUM >= 0x070c00 +#if LIBCURL_VERSION_NUM >= 0x070c00 #define curl_global_init(a) curl_global_init_mem(a, xmalloc, free, \ xrealloc, xstrdup, xcalloc) #endif @@ -37,10 +31,6 @@ #define NO_CURL_EASY_DUPHANDLE #endif -#if LIBCURL_VERSION_NUM < 0x070a03 -#define CURLE_HTTP_RETURNED_ERROR CURLE_HTTP_NOT_FOUND -#endif - #if LIBCURL_VERSION_NUM < 0x070c03 #define NO_CURL_IOCTL #endif diff --git a/remote-curl.c b/remote-curl.c index 9d432c299a..9e6918468e 100644 --- a/remote-curl.c +++ b/remote-curl.c @@ -185,8 +185,6 @@ static int set_option(const char *name, const char *value) strbuf_detach(&unquoted, NULL)); } return 0; - -#if LIBCURL_VERSION_NUM >= 0x070a08 } else if (!strcmp(name, "family")) { if (!strcmp(value, "ipv4")) git_curl_ipresolve = CURL_IPRESOLVE_V4; @@ -197,7 +195,6 @@ static int set_option(const char *name, const char *value) else return -1; return 0; -#endif /* LIBCURL_VERSION_NUM >= 0x070a08 */ } else if (!strcmp(name, "from-promisor")) { options.from_promisor = 1; return 0; From 013c7e2b070f5b69d6585b0c18426a959f1bf739 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 30 Jul 2021 11:31:54 +0200 Subject: [PATCH 049/762] http: drop support for curl < 7.16.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In the last commit we dropped support for curl < 7.11.1, let's continue that and drop support for versions older than 7.16.0. This allows us to get rid of some now-obsolete #ifdefs. Choosing 7.16.0 is a somewhat arbitrary cutoff: 1. It came out in October of 2006, almost 15 years ago. Besides being a nice round number, around 10 years is a common end-of-life support period, even for conservative distributions. 2. That version introduced the curl_multi interface, which gives us a lot of bang for the buck in removing #ifdefs RHEL 5 came with curl 7.15.5[1] (released in August 2006). RHEL 5's extended life cycle program ended on 2020-11-30[1]. RHEL 6 comes with curl 7.19.7 (released in November 2009), and RHEL 7 comes with 7.29.0 (released in February 2013). 1. http://lore.kernel.org/git/873e1f31-2a96-5b72-2f20-a5816cad1b51@jupiterrise.com Signed-off-by: Jeff King Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- http-push.c | 23 --------------------- http-walker.c | 12 ----------- http.c | 56 +-------------------------------------------------- http.h | 25 +---------------------- imap-send.c | 4 ---- remote-curl.c | 4 ---- 6 files changed, 2 insertions(+), 122 deletions(-) diff --git a/http-push.c b/http-push.c index d7cb1675a2..aa3de7c108 100644 --- a/http-push.c +++ b/http-push.c @@ -203,10 +203,8 @@ static void curl_setup_http(CURL *curl, const char *url, curl_easy_setopt(curl, CURLOPT_INFILE, buffer); curl_easy_setopt(curl, CURLOPT_INFILESIZE, buffer->buf.len); curl_easy_setopt(curl, CURLOPT_READFUNCTION, fread_buffer); -#ifndef NO_CURL_IOCTL curl_easy_setopt(curl, CURLOPT_IOCTLFUNCTION, ioctl_buffer); curl_easy_setopt(curl, CURLOPT_IOCTLDATA, buffer); -#endif curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_fn); curl_easy_setopt(curl, CURLOPT_NOBODY, 0); curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, custom_req); @@ -249,8 +247,6 @@ static void process_response(void *callback_data) finish_request(request); } -#ifdef USE_CURL_MULTI - static void start_fetch_loose(struct transfer_request *request) { struct active_request_slot *slot; @@ -299,7 +295,6 @@ static void start_mkcol(struct transfer_request *request) FREE_AND_NULL(request->url); } } -#endif static void start_fetch_packed(struct transfer_request *request) { @@ -605,7 +600,6 @@ static void finish_request(struct transfer_request *request) } } -#ifdef USE_CURL_MULTI static int is_running_queue; static int fill_active_slot(void *unused) { @@ -629,7 +623,6 @@ static int fill_active_slot(void *unused) } return 0; } -#endif static void get_remote_object_list(unsigned char parent); @@ -658,10 +651,8 @@ static void add_fetch_request(struct object *obj) request->next = request_queue_head; request_queue_head = request; -#ifdef USE_CURL_MULTI fill_active_slots(); step_active_slots(); -#endif } static int add_send_request(struct object *obj, struct remote_lock *lock) @@ -696,10 +687,8 @@ static int add_send_request(struct object *obj, struct remote_lock *lock) request->next = request_queue_head; request_queue_head = request; -#ifdef USE_CURL_MULTI fill_active_slots(); step_active_slots(); -#endif return 1; } @@ -1682,21 +1671,15 @@ static int delete_remote_branch(const char *pattern, int force) static void run_request_queue(void) { -#ifdef USE_CURL_MULTI is_running_queue = 1; fill_active_slots(); add_fill_function(NULL, fill_active_slot); -#endif do { finish_all_active_slots(); -#ifdef USE_CURL_MULTI fill_active_slots(); -#endif } while (request_queue_head && !aborted); -#ifdef USE_CURL_MULTI is_running_queue = 0; -#endif } int cmd_main(int argc, const char **argv) @@ -1770,10 +1753,6 @@ int cmd_main(int argc, const char **argv) break; } -#ifndef USE_CURL_MULTI - die("git-push is not available for http/https repository when not compiled with USE_CURL_MULTI"); -#endif - if (!repo->url) usage(http_push_usage); @@ -1786,9 +1765,7 @@ int cmd_main(int argc, const char **argv) http_init(NULL, repo->url, 1); -#ifdef USE_CURL_MULTI is_running_queue = 0; -#endif /* Verify DAV compliance/lock support */ if (!locking_available()) { diff --git a/http-walker.c b/http-walker.c index 90d8ecb57e..19e31623f0 100644 --- a/http-walker.c +++ b/http-walker.c @@ -127,7 +127,6 @@ static void release_object_request(struct object_request *obj_req) free(obj_req); } -#ifdef USE_CURL_MULTI static int fill_active_slot(struct walker *walker) { struct object_request *obj_req; @@ -146,7 +145,6 @@ static int fill_active_slot(struct walker *walker) } return 0; } -#endif static void prefetch(struct walker *walker, unsigned char *sha1) { @@ -163,10 +161,8 @@ static void prefetch(struct walker *walker, unsigned char *sha1) http_is_verbose = walker->get_verbosely; list_add_tail(&newreq->node, &object_queue_head); -#ifdef USE_CURL_MULTI fill_active_slots(); step_active_slots(); -#endif } static int is_alternate_allowed(const char *url) @@ -357,11 +353,9 @@ static void fetch_alternates(struct walker *walker, const char *base) * wait for them to arrive and return to processing this request's * curl message */ -#ifdef USE_CURL_MULTI while (cdata->got_alternates == 0) { step_active_slots(); } -#endif /* Nothing to do if they've already been fetched */ if (cdata->got_alternates == 1) @@ -505,12 +499,8 @@ static int fetch_object(struct walker *walker, unsigned char *hash) return 0; } -#ifdef USE_CURL_MULTI while (obj_req->state == WAITING) step_active_slots(); -#else - start_object_request(walker, obj_req); -#endif /* * obj_req->req might change when fetching alternates in the callback @@ -623,9 +613,7 @@ struct walker *get_http_walker(const char *url) walker->cleanup = cleanup; walker->data = data; -#ifdef USE_CURL_MULTI add_fill_function(walker, (int (*)(void *)) fill_active_slot); -#endif return walker; } diff --git a/http.c b/http.c index 56182a89e2..ef00e93023 100644 --- a/http.c +++ b/http.c @@ -26,10 +26,8 @@ ssize_t http_post_buffer = 16 * LARGE_PACKET_MAX; static int min_curl_sessions = 1; static int curl_session_count; -#ifdef USE_CURL_MULTI static int max_requests = -1; static CURLM *curlm; -#endif #ifndef NO_CURL_EASY_DUPHANDLE static CURL *curl_default; #endif @@ -117,14 +115,6 @@ static int curl_empty_auth = -1; enum http_follow_config http_follow_config = HTTP_FOLLOW_INITIAL; -#if LIBCURL_VERSION_NUM >= 0x071700 -/* Use CURLOPT_KEYPASSWD as is */ -#elif LIBCURL_VERSION_NUM >= 0x070903 -#define CURLOPT_KEYPASSWD CURLOPT_SSLKEYPASSWD -#else -#define CURLOPT_KEYPASSWD CURLOPT_SSLCERTPASSWD -#endif - static struct credential cert_auth = CREDENTIAL_INIT; static int ssl_cert_password_required; static unsigned long http_auth_methods = CURLAUTH_ANY; @@ -168,7 +158,6 @@ size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) return size / eltsize; } -#ifndef NO_CURL_IOCTL curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp) { struct buffer *buffer = clientp; @@ -185,7 +174,6 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp) return CURLIOE_UNKNOWNCMD; } } -#endif size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) { @@ -233,12 +221,9 @@ static void finish_active_slot(struct active_request_slot *slot) static void xmulti_remove_handle(struct active_request_slot *slot) { -#ifdef USE_CURL_MULTI curl_multi_remove_handle(curlm, slot->curl); -#endif } -#ifdef USE_CURL_MULTI static void process_curl_messages(void) { int num_messages; @@ -266,7 +251,6 @@ static void process_curl_messages(void) curl_message = curl_multi_info_read(curlm, &num_messages); } } -#endif static int http_options(const char *var, const char *value, void *cb) { @@ -315,18 +299,14 @@ static int http_options(const char *var, const char *value, void *cb) if (!strcmp("http.minsessions", var)) { min_curl_sessions = git_config_int(var, value); -#ifndef USE_CURL_MULTI if (min_curl_sessions > 1) min_curl_sessions = 1; -#endif return 0; } -#ifdef USE_CURL_MULTI if (!strcmp("http.maxrequests", var)) { max_requests = git_config_int(var, value); return 0; } -#endif if (!strcmp("http.lowspeedlimit", var)) { curl_low_speed_limit = (long)git_config_int(var, value); return 0; @@ -574,7 +554,7 @@ static void set_curl_keepalive(CURL *c) curl_easy_setopt(c, CURLOPT_TCP_KEEPALIVE, 1); } -#elif LIBCURL_VERSION_NUM >= 0x071000 +#else static int sockopt_callback(void *client, curl_socket_t fd, curlsocktype type) { int ka = 1; @@ -595,12 +575,6 @@ static void set_curl_keepalive(CURL *c) { curl_easy_setopt(c, CURLOPT_SOCKOPTFUNCTION, sockopt_callback); } - -#else -static void set_curl_keepalive(CURL *c) -{ - /* not supported on older curl versions */ -} #endif static void redact_sensitive_header(struct strbuf *header) @@ -1121,7 +1095,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth) no_pragma_header = curl_slist_append(http_copy_default_headers(), "Pragma:"); -#ifdef USE_CURL_MULTI { char *http_max_requests = getenv("GIT_HTTP_MAX_REQUESTS"); if (http_max_requests != NULL) @@ -1131,7 +1104,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth) curlm = curl_multi_init(); if (!curlm) die("curl_multi_init failed"); -#endif if (getenv("GIT_SSL_NO_VERIFY")) curl_ssl_verify = 0; @@ -1154,10 +1126,8 @@ void http_init(struct remote *remote, const char *url, int proactive_auth) curl_ssl_verify = 1; curl_session_count = 0; -#ifdef USE_CURL_MULTI if (max_requests < 1) max_requests = DEFAULT_MAX_REQUESTS; -#endif set_from_env(&http_proxy_ssl_cert, "GIT_PROXY_SSL_CERT"); set_from_env(&http_proxy_ssl_key, "GIT_PROXY_SSL_KEY"); @@ -1201,9 +1171,7 @@ void http_cleanup(void) curl_easy_cleanup(curl_default); #endif -#ifdef USE_CURL_MULTI curl_multi_cleanup(curlm); -#endif curl_global_cleanup(); string_list_clear(&extra_http_headers, 0); @@ -1250,7 +1218,6 @@ struct active_request_slot *get_active_slot(void) struct active_request_slot *slot = active_queue_head; struct active_request_slot *newslot; -#ifdef USE_CURL_MULTI int num_transfers; /* Wait for a slot to open up if the queue is full */ @@ -1259,7 +1226,6 @@ struct active_request_slot *get_active_slot(void) if (num_transfers < active_requests) process_curl_messages(); } -#endif while (slot != NULL && slot->in_use) slot = slot->next; @@ -1330,7 +1296,6 @@ struct active_request_slot *get_active_slot(void) int start_active_slot(struct active_request_slot *slot) { -#ifdef USE_CURL_MULTI CURLMcode curlm_result = curl_multi_add_handle(curlm, slot->curl); int num_transfers; @@ -1348,11 +1313,9 @@ int start_active_slot(struct active_request_slot *slot) * something. */ curl_multi_perform(curlm, &num_transfers); -#endif return 1; } -#ifdef USE_CURL_MULTI struct fill_chain { void *data; int (*fill)(void *); @@ -1411,11 +1374,9 @@ void step_active_slots(void) fill_active_slots(); } } -#endif void run_active_slot(struct active_request_slot *slot) { -#ifdef USE_CURL_MULTI fd_set readfds; fd_set writefds; fd_set excfds; @@ -1428,7 +1389,6 @@ void run_active_slot(struct active_request_slot *slot) step_active_slots(); if (slot->in_use) { -#if LIBCURL_VERSION_NUM >= 0x070f04 long curl_timeout; curl_multi_timeout(curlm, &curl_timeout); if (curl_timeout == 0) { @@ -1440,10 +1400,6 @@ void run_active_slot(struct active_request_slot *slot) select_timeout.tv_sec = curl_timeout / 1000; select_timeout.tv_usec = (curl_timeout % 1000) * 1000; } -#else - select_timeout.tv_sec = 0; - select_timeout.tv_usec = 50000; -#endif max_fd = -1; FD_ZERO(&readfds); @@ -1466,12 +1422,6 @@ void run_active_slot(struct active_request_slot *slot) select(max_fd+1, &readfds, &writefds, &excfds, &select_timeout); } } -#else - while (slot->in_use) { - slot->curl_result = curl_easy_perform(slot->curl); - finish_active_slot(slot); - } -#endif } static void release_active_slot(struct active_request_slot *slot) @@ -1485,9 +1435,7 @@ static void release_active_slot(struct active_request_slot *slot) curl_session_count--; } } -#ifdef USE_CURL_MULTI fill_active_slots(); -#endif } void finish_all_active_slots(void) @@ -1613,12 +1561,10 @@ static int handle_curl_result(struct slot_results *results) } else { if (results->http_connectcode == 407) credential_reject(&proxy_auth); -#if LIBCURL_VERSION_NUM >= 0x070c00 if (!curl_errorstr[0]) strlcpy(curl_errorstr, curl_easy_strerror(results->curl_result), sizeof(curl_errorstr)); -#endif return HTTP_ERROR; } } diff --git a/http.h b/http.h index d2f8cc5661..cb092622a7 100644 --- a/http.h +++ b/http.h @@ -10,31 +10,12 @@ #include "remote.h" #include "url.h" -/* - * We detect based on the cURL version if multi-transfer is - * usable in this implementation and define this symbol accordingly. - * This shouldn't be set by the Makefile or by the user (e.g. via CFLAGS). - */ -#undef USE_CURL_MULTI - -#if LIBCURL_VERSION_NUM >= 0x071000 -#define USE_CURL_MULTI #define DEFAULT_MAX_REQUESTS 5 -#endif -#if LIBCURL_VERSION_NUM >= 0x070c00 -#define curl_global_init(a) curl_global_init_mem(a, xmalloc, free, \ - xrealloc, xstrdup, xcalloc) -#endif - -#if (LIBCURL_VERSION_NUM < 0x070c04) || (LIBCURL_VERSION_NUM == 0x071000) +#if LIBCURL_VERSION_NUM == 0x071000 #define NO_CURL_EASY_DUPHANDLE #endif -#if LIBCURL_VERSION_NUM < 0x070c03 -#define NO_CURL_IOCTL -#endif - /* * CURLOPT_USE_SSL was known as CURLOPT_FTP_SSL up to 7.16.4, * and the constants were known as CURLFTPSSL_* @@ -72,9 +53,7 @@ struct buffer { size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *strbuf); size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *strbuf); size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf); -#ifndef NO_CURL_IOCTL curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp); -#endif /* Slot lifecycle functions */ struct active_request_slot *get_active_slot(void); @@ -91,11 +70,9 @@ void finish_all_active_slots(void); int run_one_slot(struct active_request_slot *slot, struct slot_results *results); -#ifdef USE_CURL_MULTI void fill_active_slots(void); void add_fill_function(void *data, int (*fill)(void *)); void step_active_slots(void); -#endif void http_init(struct remote *remote, const char *url, int proactive_auth); diff --git a/imap-send.c b/imap-send.c index bb085d66d1..9844328b7b 100644 --- a/imap-send.c +++ b/imap-send.c @@ -1526,11 +1526,7 @@ static int curl_append_msgs_to_imap(struct imap_server_conf *server, if (cred.username) { if (res == CURLE_OK) credential_approve(&cred); -#if LIBCURL_VERSION_NUM >= 0x070d01 else if (res == CURLE_LOGIN_DENIED) -#else - else -#endif credential_reject(&cred); } diff --git a/remote-curl.c b/remote-curl.c index 9e6918468e..482d5a4656 100644 --- a/remote-curl.c +++ b/remote-curl.c @@ -706,7 +706,6 @@ static size_t rpc_out(void *ptr, size_t eltsize, return avail; } -#ifndef NO_CURL_IOCTL static curlioerr rpc_ioctl(CURL *handle, int cmd, void *clientp) { struct rpc_state *rpc = clientp; @@ -727,7 +726,6 @@ static curlioerr rpc_ioctl(CURL *handle, int cmd, void *clientp) return CURLIOE_UNKNOWNCMD; } } -#endif struct check_pktline_state { char len_buf[4]; @@ -946,10 +944,8 @@ retry: rpc->initial_buffer = 1; curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out); curl_easy_setopt(slot->curl, CURLOPT_INFILE, rpc); -#ifndef NO_CURL_IOCTL curl_easy_setopt(slot->curl, CURLOPT_IOCTLFUNCTION, rpc_ioctl); curl_easy_setopt(slot->curl, CURLOPT_IOCTLDATA, rpc); -#endif if (options.verbosity > 1) { fprintf(stderr, "POST %s (chunked)\n", rpc->service_name); fflush(stderr); From 644de29e220de7441e51a29bf512278b3de0bbe1 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 30 Jul 2021 19:59:44 +0200 Subject: [PATCH 050/762] http: drop support for curl < 7.19.4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In the last commit we dropped support for curl < 7.16.0, let's continue that and drop support for versions older than 7.19.3. This allows us to simplify the code by getting rid of some "#ifdef"'s. Git was broken with vanilla curl < 7.19.4 from v2.12.0 until v2.15.0. Compiling with it was broken by using CURLPROTO_* outside any "#ifdef" in aeae4db174 (http: create function to get curl allowed protocols, 2016-12-14), and fixed in v2.15.0 in f18777ba6ef (http: fix handling of missing CURLPROTO_*, 2017-08-11). It's unclear how much anyone was impacted by that in practice, since as noted in [1] RHEL versions using curl older than that still compiled, because RedHat backported some features. Perhaps other vendors did the same. Still, it's one datapoint indicating that it wasn't in active use at the time. That (the v2.12.0 release) was in Feb 24, 2017, with v2.15.0 on Oct 30, 2017, it's now mid-2021. 1. http://lore.kernel.org/git/c8a2716d-76ac-735c-57f9-175ca3acbcb0@jupiterrise.com; followed-up by f18777ba6ef (http: fix handling of missing CURLPROTO_*, 2017-08-11) Signed-off-by: Jeff King Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- http.c | 50 -------------------------------------------------- http.h | 4 ---- 2 files changed, 54 deletions(-) diff --git a/http.c b/http.c index ef00e93023..1f0d7664d3 100644 --- a/http.c +++ b/http.c @@ -28,9 +28,7 @@ static int min_curl_sessions = 1; static int curl_session_count; static int max_requests = -1; static CURLM *curlm; -#ifndef NO_CURL_EASY_DUPHANDLE static CURL *curl_default; -#endif #define PREV_BUF_SIZE 4096 @@ -440,24 +438,8 @@ static void init_curl_http_auth(CURL *result) credential_fill(&http_auth); -#if LIBCURL_VERSION_NUM >= 0x071301 curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username); curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password); -#else - { - static struct strbuf up = STRBUF_INIT; - /* - * Note that we assume we only ever have a single set of - * credentials in a given program run, so we do not have - * to worry about updating this buffer, only setting its - * initial value. - */ - if (!up.len) - strbuf_addf(&up, "%s:%s", - http_auth.username, http_auth.password); - curl_easy_setopt(result, CURLOPT_USERPWD, up.buf); - } -#endif } /* *var must be free-able */ @@ -471,22 +453,10 @@ static void var_override(const char **var, char *value) static void set_proxyauth_name_password(CURL *result) { -#if LIBCURL_VERSION_NUM >= 0x071301 curl_easy_setopt(result, CURLOPT_PROXYUSERNAME, proxy_auth.username); curl_easy_setopt(result, CURLOPT_PROXYPASSWORD, proxy_auth.password); -#else - struct strbuf s = STRBUF_INIT; - - strbuf_addstr_urlencode(&s, proxy_auth.username, - is_rfc3986_unreserved); - strbuf_addch(&s, ':'); - strbuf_addstr_urlencode(&s, proxy_auth.password, - is_rfc3986_unreserved); - curl_proxyuserpwd = strbuf_detach(&s, NULL); - curl_easy_setopt(result, CURLOPT_PROXYUSERPWD, curl_proxyuserpwd); -#endif } static void init_curl_proxy_auth(CURL *result) @@ -748,7 +718,6 @@ void setup_curl_trace(CURL *handle) curl_easy_setopt(handle, CURLOPT_DEBUGDATA, NULL); } -#ifdef CURLPROTO_HTTP static long get_curl_allowed_protocols(int from_user) { long allowed_protocols = 0; @@ -764,7 +733,6 @@ static long get_curl_allowed_protocols(int from_user) return allowed_protocols; } -#endif #if LIBCURL_VERSION_NUM >=0x072f00 static int get_curl_http_version_opt(const char *version_string, long *opt) @@ -906,19 +874,11 @@ static CURL *get_curl_handle(void) } curl_easy_setopt(result, CURLOPT_MAXREDIRS, 20); -#if LIBCURL_VERSION_NUM >= 0x071301 curl_easy_setopt(result, CURLOPT_POSTREDIR, CURL_REDIR_POST_ALL); -#elif LIBCURL_VERSION_NUM >= 0x071101 - curl_easy_setopt(result, CURLOPT_POST301, 1); -#endif -#ifdef CURLPROTO_HTTP curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, get_curl_allowed_protocols(0)); curl_easy_setopt(result, CURLOPT_PROTOCOLS, get_curl_allowed_protocols(-1)); -#else - warning(_("Protocol restrictions not supported with cURL < 7.19.4")); -#endif if (getenv("GIT_CURL_VERBOSE")) http_trace_curl_no_data(); setup_curl_trace(result); @@ -1012,11 +972,9 @@ static CURL *get_curl_handle(void) die("Invalid proxy URL '%s'", curl_http_proxy); curl_easy_setopt(result, CURLOPT_PROXY, proxy_auth.host); -#if LIBCURL_VERSION_NUM >= 0x071304 var_override(&curl_no_proxy, getenv("NO_PROXY")); var_override(&curl_no_proxy, getenv("no_proxy")); curl_easy_setopt(result, CURLOPT_NOPROXY, curl_no_proxy); -#endif } init_curl_proxy_auth(result); @@ -1147,9 +1105,7 @@ void http_init(struct remote *remote, const char *url, int proactive_auth) ssl_cert_password_required = 1; } -#ifndef NO_CURL_EASY_DUPHANDLE curl_default = get_curl_handle(); -#endif } void http_cleanup(void) @@ -1167,9 +1123,7 @@ void http_cleanup(void) } active_queue_head = NULL; -#ifndef NO_CURL_EASY_DUPHANDLE curl_easy_cleanup(curl_default); -#endif curl_multi_cleanup(curlm); curl_global_cleanup(); @@ -1248,11 +1202,7 @@ struct active_request_slot *get_active_slot(void) } if (slot->curl == NULL) { -#ifdef NO_CURL_EASY_DUPHANDLE - slot->curl = get_curl_handle(); -#else slot->curl = curl_easy_duphandle(curl_default); -#endif curl_session_count++; } diff --git a/http.h b/http.h index cb092622a7..19f19dbe74 100644 --- a/http.h +++ b/http.h @@ -12,10 +12,6 @@ #define DEFAULT_MAX_REQUESTS 5 -#if LIBCURL_VERSION_NUM == 0x071000 -#define NO_CURL_EASY_DUPHANDLE -#endif - /* * CURLOPT_USE_SSL was known as CURLOPT_FTP_SSL up to 7.16.4, * and the constants were known as CURLFTPSSL_* From 5db9d383590c37272a9f16464a66dfbd3a3c8aff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Fri, 30 Jul 2021 19:59:45 +0200 Subject: [PATCH 051/762] http: drop support for curl < 7.19.3 and < 7.17.0 (again) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove the conditional use of CURLAUTH_DIGEST_IE and CURLOPT_USE_SSL. These two have been split from earlier simpler checks against LIBCURL_VERSION_NUM for ease of review. According to https://github.com/curl/curl/blob/master/docs/libcurl/symbols-in-versions the CURLAUTH_DIGEST_IE flag became available in 7.19.3, and CURLOPT_USE_SSL in 7.17.0. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- http.c | 4 ---- http.h | 9 --------- 2 files changed, 13 deletions(-) diff --git a/http.c b/http.c index 1f0d7664d3..e9446850a6 100644 --- a/http.c +++ b/http.c @@ -120,9 +120,7 @@ static int http_auth_methods_restricted; /* Modes for which empty_auth cannot actually help us. */ static unsigned long empty_auth_useless = CURLAUTH_BASIC -#ifdef CURLAUTH_DIGEST_IE | CURLAUTH_DIGEST_IE -#endif | CURLAUTH_DIGEST; static struct curl_slist *pragma_header; @@ -893,10 +891,8 @@ static CURL *get_curl_handle(void) if (curl_ftp_no_epsv) curl_easy_setopt(result, CURLOPT_FTP_USE_EPSV, 0); -#ifdef CURLOPT_USE_SSL if (curl_ssl_try) curl_easy_setopt(result, CURLOPT_USE_SSL, CURLUSESSL_TRY); -#endif /* * CURL also examines these variables as a fallback; but we need to query diff --git a/http.h b/http.h index 19f19dbe74..3db5a0cf32 100644 --- a/http.h +++ b/http.h @@ -12,15 +12,6 @@ #define DEFAULT_MAX_REQUESTS 5 -/* - * CURLOPT_USE_SSL was known as CURLOPT_FTP_SSL up to 7.16.4, - * and the constants were known as CURLFTPSSL_* -*/ -#if !defined(CURLOPT_USE_SSL) && defined(CURLOPT_FTP_SSL) -#define CURLOPT_USE_SSL CURLOPT_FTP_SSL -#define CURLUSESSL_TRY CURLFTPSSL_TRY -#endif - struct slot_results { CURLcode curl_result; long http_code; From 8dda4cbdf258be05e7201ef5d6e9d532cacb9178 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Fri, 30 Jul 2021 19:59:46 +0200 Subject: [PATCH 052/762] http: rename CURLOPT_FILE to CURLOPT_WRITEDATA MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CURLOPT_FILE name is an alias for CURLOPT_WRITEDATA, the CURLOPT_WRITEDATA name has been preferred since curl 7.9.7, released in May 2002[1]. 1. https://curl.se/libcurl/c/CURLOPT_WRITEDATA.html Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- http-push.c | 6 +++--- http-walker.c | 2 +- http.c | 6 +++--- remote-curl.c | 4 ++-- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/http-push.c b/http-push.c index aa3de7c108..3309aaf004 100644 --- a/http-push.c +++ b/http-push.c @@ -883,7 +883,7 @@ static struct remote_lock *lock_remote(const char *path, long timeout) slot->results = &results; curl_setup_http(slot->curl, url, DAV_LOCK, &out_buffer, fwrite_buffer); curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers); - curl_easy_setopt(slot->curl, CURLOPT_FILE, &in_buffer); + curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer); CALLOC_ARRAY(lock, 1); lock->timeout = -1; @@ -1142,7 +1142,7 @@ static void remote_ls(const char *path, int flags, curl_setup_http(slot->curl, url, DAV_PROPFIND, &out_buffer, fwrite_buffer); curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers); - curl_easy_setopt(slot->curl, CURLOPT_FILE, &in_buffer); + curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer); if (start_active_slot(slot)) { run_active_slot(slot); @@ -1216,7 +1216,7 @@ static int locking_available(void) curl_setup_http(slot->curl, repo->url, DAV_PROPFIND, &out_buffer, fwrite_buffer); curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers); - curl_easy_setopt(slot->curl, CURLOPT_FILE, &in_buffer); + curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer); if (start_active_slot(slot)) { run_active_slot(slot); diff --git a/http-walker.c b/http-walker.c index 19e31623f0..910fae539b 100644 --- a/http-walker.c +++ b/http-walker.c @@ -378,7 +378,7 @@ static void fetch_alternates(struct walker *walker, const char *base) alt_req.walker = walker; slot->callback_data = &alt_req; - curl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer); + curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &buffer); curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer); curl_easy_setopt(slot->curl, CURLOPT_URL, url.buf); diff --git a/http.c b/http.c index e9446850a6..a0f169d2fe 100644 --- a/http.c +++ b/http.c @@ -1769,7 +1769,7 @@ static int http_request(const char *url, curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 1); } else { curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0); - curl_easy_setopt(slot->curl, CURLOPT_FILE, result); + curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, result); if (target == HTTP_REQUEST_FILE) { off_t posn = ftello(result); @@ -2186,7 +2186,7 @@ struct http_pack_request *new_direct_http_pack_request( } preq->slot = get_active_slot(); - curl_easy_setopt(preq->slot->curl, CURLOPT_FILE, preq->packfile); + curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEDATA, preq->packfile); curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite); curl_easy_setopt(preq->slot->curl, CURLOPT_URL, preq->url); curl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER, @@ -2357,7 +2357,7 @@ struct http_object_request *new_http_object_request(const char *base_url, freq->slot = get_active_slot(); - curl_easy_setopt(freq->slot->curl, CURLOPT_FILE, freq); + curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEDATA, freq); curl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0); curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file); curl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr); diff --git a/remote-curl.c b/remote-curl.c index 482d5a4656..bf795f90c6 100644 --- a/remote-curl.c +++ b/remote-curl.c @@ -853,7 +853,7 @@ static int probe_rpc(struct rpc_state *rpc, struct slot_results *results) curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, 4); curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers); curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer); - curl_easy_setopt(slot->curl, CURLOPT_FILE, &buf); + curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &buf); err = run_slot(slot, results); @@ -1016,7 +1016,7 @@ retry: rpc_in_data.slot = slot; rpc_in_data.check_pktline = stateless_connect; memset(&rpc_in_data.pktline_state, 0, sizeof(rpc_in_data.pktline_state)); - curl_easy_setopt(slot->curl, CURLOPT_FILE, &rpc_in_data); + curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &rpc_in_data); curl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 0); From e6b0a8fab8e5a29f11faef898905e0b3c700e683 Mon Sep 17 00:00:00 2001 From: Han-Wen Nienhuys Date: Mon, 2 Aug 2021 16:53:26 +0000 Subject: [PATCH 053/762] t6050: use git-update-ref rather than filesystem access Signed-off-by: Han-Wen Nienhuys Signed-off-by: Junio C Hamano --- t/t6050-replace.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t6050-replace.sh b/t/t6050-replace.sh index e33d512ec1..2500acc2ef 100755 --- a/t/t6050-replace.sh +++ b/t/t6050-replace.sh @@ -132,7 +132,7 @@ tagger T A Gger <> 0 +0000 EOF test_expect_success 'tag replaced commit' ' - git mktag .git/refs/tags/mytag + git update-ref refs/tags/mytag $(git mktag Date: Mon, 2 Aug 2021 16:53:27 +0000 Subject: [PATCH 054/762] t1503: mark symlink test as REFFILES Signed-off-by: Han-Wen Nienhuys Signed-off-by: Junio C Hamano --- t/t1503-rev-parse-verify.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t1503-rev-parse-verify.sh b/t/t1503-rev-parse-verify.sh index bf08102391..40958615eb 100755 --- a/t/t1503-rev-parse-verify.sh +++ b/t/t1503-rev-parse-verify.sh @@ -142,7 +142,7 @@ test_expect_success 'main@{n} for various n' ' test_must_fail git rev-parse --verify main@{$Np1} ' -test_expect_success SYMLINKS 'ref resolution not confused by broken symlinks' ' +test_expect_success SYMLINKS,REFFILES 'ref resolution not confused by broken symlinks' ' ln -s does-not-exist .git/refs/heads/broken && test_must_fail git rev-parse --verify broken ' From e46775cf9e1c6ddca0ada3c1ca24876b79a23d9a Mon Sep 17 00:00:00 2001 From: Han-Wen Nienhuys Date: Mon, 2 Aug 2021 16:53:28 +0000 Subject: [PATCH 055/762] t6120: use git-update-ref rather than filesystem access Signed-off-by: Han-Wen Nienhuys Signed-off-by: Junio C Hamano --- t/t6120-describe.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/t/t6120-describe.sh b/t/t6120-describe.sh index 1a501ee09e..bae2419150 100755 --- a/t/t6120-describe.sh +++ b/t/t6120-describe.sh @@ -107,7 +107,8 @@ test_expect_success 'describe --contains defaults to HEAD without commit-ish' ' check_describe tags/A --all A^0 test_expect_success 'renaming tag A to Q locally produces a warning' " - mv .git/refs/tags/A .git/refs/tags/Q && + git update-ref refs/tags/Q $(git rev-parse refs/tags/A) && + git update-ref -d refs/tags/A && git describe HEAD 2>err >out && cat >expected <<-\EOF && warning: tag 'Q' is externally known as 'A' @@ -135,7 +136,8 @@ test_expect_success 'abbrev=0 will not break misplaced tag (2)' ' ' test_expect_success 'rename tag Q back to A' ' - mv .git/refs/tags/Q .git/refs/tags/A + git update-ref refs/tags/A $(git rev-parse refs/tags/Q) && + git update-ref -d refs/tags/Q ' test_expect_success 'pack tag refs' 'git pack-refs' From 2f566d665ac3ef97715617aae839fa2e6f7fce93 Mon Sep 17 00:00:00 2001 From: Han-Wen Nienhuys Date: Mon, 2 Aug 2021 16:53:29 +0000 Subject: [PATCH 056/762] t3320: use git-symbolic-ref rather than filesystem access Signed-off-by: Han-Wen Nienhuys Signed-off-by: Junio C Hamano --- t/t3320-notes-merge-worktrees.sh | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/t/t3320-notes-merge-worktrees.sh b/t/t3320-notes-merge-worktrees.sh index 052516e6c6..6b2d507f3e 100755 --- a/t/t3320-notes-merge-worktrees.sh +++ b/t/t3320-notes-merge-worktrees.sh @@ -46,8 +46,9 @@ test_expect_success 'create some new worktrees' ' test_expect_success 'merge z into y fails and sets NOTES_MERGE_REF' ' git config core.notesRef refs/notes/y && test_must_fail git notes merge z && - echo "ref: refs/notes/y" >expect && - test_cmp expect .git/NOTES_MERGE_REF + echo "refs/notes/y" >expect && + git symbolic-ref NOTES_MERGE_REF >actual && + test_cmp expect actual ' test_expect_success 'merge z into y while mid-merge in another workdir fails' ' @@ -57,7 +58,7 @@ test_expect_success 'merge z into y while mid-merge in another workdir fails' ' test_must_fail git notes merge z 2>err && test_i18ngrep "a notes merge into refs/notes/y is already in-progress at" err ) && - test_path_is_missing .git/worktrees/worktree/NOTES_MERGE_REF + test_must_fail git -C worktree symbolic-ref NOTES_MERGE_REF ' test_expect_success 'merge z into x while mid-merge on y succeeds' ' @@ -68,8 +69,9 @@ test_expect_success 'merge z into x while mid-merge on y succeeds' ' test_i18ngrep "Automatic notes merge failed" out && grep -v "A notes merge into refs/notes/x is already in-progress in" out ) && - echo "ref: refs/notes/x" >expect && - test_cmp expect .git/worktrees/worktree2/NOTES_MERGE_REF + echo "refs/notes/x" >expect && + git -C worktree2 symbolic-ref NOTES_MERGE_REF >actual && + test_cmp expect actual ' test_done From 100ac47bf364290768001b2b438a84993a7042f6 Mon Sep 17 00:00:00 2001 From: Han-Wen Nienhuys Date: Mon, 2 Aug 2021 16:53:30 +0000 Subject: [PATCH 057/762] t2402: use ref-store test helper to create broken symlink Signed-off-by: Han-Wen Nienhuys Signed-off-by: Junio C Hamano --- t/t2402-worktree-list.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t2402-worktree-list.sh b/t/t2402-worktree-list.sh index fedcefe8de..4012bd67b0 100755 --- a/t/t2402-worktree-list.sh +++ b/t/t2402-worktree-list.sh @@ -230,7 +230,7 @@ test_expect_success 'broken main worktree still at the top' ' EOF cd linked && echo "worktree $(pwd)" >expected && - echo "ref: .broken" >../.git/HEAD && + (cd ../ && test-tool ref-store main create-symref HEAD .broken ) && git worktree list --porcelain >out && head -n 3 out >actual && test_cmp ../expected actual && From ace40eab9e8ee4d087b8ba7b603db4958e8ad28c Mon Sep 17 00:00:00 2001 From: Han-Wen Nienhuys Date: Mon, 2 Aug 2021 16:53:31 +0000 Subject: [PATCH 058/762] t1405: use 'git reflog exists' to check reflog existence This fixes a test failure for reftable. Signed-off-by: Han-Wen Nienhuys Signed-off-by: Junio C Hamano --- t/t1405-main-ref-store.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/t/t1405-main-ref-store.sh b/t/t1405-main-ref-store.sh index a237d9880e..92b0487324 100755 --- a/t/t1405-main-ref-store.sh +++ b/t/t1405-main-ref-store.sh @@ -98,12 +98,12 @@ test_expect_success 'reflog_exists(HEAD)' ' test_expect_success 'delete_reflog(HEAD)' ' $RUN delete-reflog HEAD && - ! test -f .git/logs/HEAD + test_must_fail git reflog exists HEAD ' test_expect_success 'create-reflog(HEAD)' ' $RUN create-reflog HEAD 1 && - test -f .git/logs/HEAD + git reflog exists HEAD ' test_expect_success 'delete_ref(refs/heads/foo)' ' From a50234b3be9d6ff75dfca9b0802e7cf97179ff11 Mon Sep 17 00:00:00 2001 From: Han-Wen Nienhuys Date: Mon, 2 Aug 2021 16:53:32 +0000 Subject: [PATCH 059/762] t1405: mark test for 'git pack-refs' as REFFILES The tests verifies that "pack-refs" causes loose refs to be packed. As both loose and packed refs are concepts specific to the files backend, mark the test as REFFILES. Check the outcome of the pack-refs operation. This was apparently forgotten in the commit introducing this test: 16feb99d (Mar 26 2017, "t1405: some basic tests on main ref store"). Signed-off-by: Han-Wen Nienhuys Signed-off-by: Junio C Hamano --- t/t1405-main-ref-store.sh | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/t/t1405-main-ref-store.sh b/t/t1405-main-ref-store.sh index 92b0487324..49718b7ea7 100755 --- a/t/t1405-main-ref-store.sh +++ b/t/t1405-main-ref-store.sh @@ -9,12 +9,18 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME RUN="test-tool ref-store main" -test_expect_success 'pack_refs(PACK_REFS_ALL | PACK_REFS_PRUNE)' ' - test_commit one && + +test_expect_success 'setup' ' + test_commit one +' + +test_expect_success REFFILES 'pack_refs(PACK_REFS_ALL | PACK_REFS_PRUNE)' ' N=`find .git/refs -type f | wc -l` && test "$N" != 0 && - $RUN pack-refs 3 && - N=`find .git/refs -type f | wc -l` + ALL_OR_PRUNE_FLAG=3 && + $RUN pack-refs ${ALL_OR_PRUNE_FLAG} && + N=`find .git/refs -type f` && + test -z "$N" ' test_expect_success 'create_symref(FOO, refs/heads/main)' ' From fe1443152689fcecbbaf0213a7bebcc23ee24a37 Mon Sep 17 00:00:00 2001 From: Han-Wen Nienhuys Date: Mon, 2 Aug 2021 16:53:33 +0000 Subject: [PATCH 060/762] t1410: mark test as REFFILES This test takes a lock on the target of a symref, and then verifies that it is possible to expire the symref's reflog. In reftable, one can only take a global lock (which would prevent the symref reflog from being expired altogether.) Signed-off-by: Han-Wen Nienhuys Signed-off-by: Junio C Hamano --- t/t1410-reflog.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/t/t1410-reflog.sh b/t/t1410-reflog.sh index 27b9080251..d42f067ff8 100755 --- a/t/t1410-reflog.sh +++ b/t/t1410-reflog.sh @@ -374,7 +374,9 @@ test_expect_failure 'reflog with non-commit entries displays all entries' ' test_line_count = 3 actual ' -test_expect_success 'reflog expire operates on symref not referrent' ' +# This test takes a lock on an individual ref; this is not supported in +# reftable. +test_expect_success REFFILES 'reflog expire operates on symref not referrent' ' git branch --create-reflog the_symref && git branch --create-reflog referrent && git update-ref referrent HEAD && From 2cf9f0fca1ad0fedb88f63f59b311b4d80b28f0e Mon Sep 17 00:00:00 2001 From: Han-Wen Nienhuys Date: Mon, 2 Aug 2021 16:53:34 +0000 Subject: [PATCH 061/762] t7064: use update-ref -d to remove upstream branch The previous code tested this by writing $ZERO_OID explicitly in the packed-refs file. This is a type of corruption that doesn't reflect realistic use-cases. In addition, even the ref-store test-tool refuses to write invalid OIDs. (update-ref interprets $ZERO_OID is deleting the ref). Signed-off-by: Han-Wen Nienhuys Signed-off-by: Junio C Hamano --- t/t7064-wtstatus-pv2.sh | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/t/t7064-wtstatus-pv2.sh b/t/t7064-wtstatus-pv2.sh index 4613882caf..eeb0534163 100755 --- a/t/t7064-wtstatus-pv2.sh +++ b/t/t7064-wtstatus-pv2.sh @@ -373,10 +373,7 @@ test_expect_success 'verify upstream fields in branch header' ' ## Test upstream-gone case. Fake this by pointing ## origin/initial-branch at a non-existing commit. - OLD=$(git rev-parse origin/initial-branch) && - NEW=$ZERO_OID && - mv .git/packed-refs .git/old-packed-refs && - sed "s/$OLD/$NEW/g" <.git/old-packed-refs >.git/packed-refs && + git update-ref -d refs/remotes/origin/initial-branch && HUF=$(git rev-parse HEAD) && From f95661b74088f019847ee8408c8ce82012089e2e Mon Sep 17 00:00:00 2001 From: Han-Wen Nienhuys Date: Mon, 2 Aug 2021 16:53:35 +0000 Subject: [PATCH 062/762] t6500: use "ls -1" to snapshot ref database state By doing ls -1 .git/{reftable,refs/heads}, we can capture changes to both reftable and packed/loose ref storage. This relies on the fact that git-pack-refs (which we're looking for here) changes the number (loose/packed storage) and/or names (reftable) files used for ref storage. Signed-off-by: Han-Wen Nienhuys Signed-off-by: Junio C Hamano --- t/t6500-gc.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh index 10c7ae7f09..c2021267f2 100755 --- a/t/t6500-gc.sh +++ b/t/t6500-gc.sh @@ -241,7 +241,7 @@ test_expect_success 'background auto gc respects lock for all operations' ' # create a ref whose loose presence we can use to detect a pack-refs run git update-ref refs/heads/should-be-loose HEAD && - test_path_is_file .git/refs/heads/should-be-loose && + (ls -1 .git/refs/heads .git/reftable >expect || true) && # now fake a concurrent gc that holds the lock; we can use our # shell pid so that it looks valid. @@ -258,7 +258,8 @@ test_expect_success 'background auto gc respects lock for all operations' ' # our gc should exit zero without doing anything run_and_wait_for_auto_gc && - test_path_is_file .git/refs/heads/should-be-loose + (ls -1 .git/refs/heads .git/reftable >actual || true) && + test_cmp expect actual ' # DO NOT leave a detached auto gc process running near the end of the From 977f8acefdbb5a7a39af52c82d21a9964f6ff7b0 Mon Sep 17 00:00:00 2001 From: Han-Wen Nienhuys Date: Mon, 2 Aug 2021 16:53:36 +0000 Subject: [PATCH 063/762] t6001: avoid direct file system access Signed-off-by: Han-Wen Nienhuys Signed-off-by: Junio C Hamano --- t/t6001-rev-list-graft.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/t/t6001-rev-list-graft.sh b/t/t6001-rev-list-graft.sh index 90d93f77fa..7294147334 100755 --- a/t/t6001-rev-list-graft.sh +++ b/t/t6001-rev-list-graft.sh @@ -23,7 +23,8 @@ test_expect_success setup ' git commit -a -m "Third in one history." && A2=$(git rev-parse --verify HEAD) && - rm -f .git/refs/heads/main .git/index && + git update-ref -d refs/heads/main && + rm -f .git/index && echo >fileA fileA again && echo >subdir/fileB fileB again && From 9ab80dd6aeadf398de45a6add5fc7ed6b2e166b7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Sat, 31 Jul 2021 10:23:05 +0200 Subject: [PATCH 064/762] bundle doc: elaborate on object prerequisites MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Split out the discussion bout "object prerequisites" into its own section, and add some more examples of the common cases. See 2e0afafebd (Add git-bundle: move objects and references by archive, 2007-02-22) for the introduction of the documentation being changed here. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Documentation/git-bundle.txt | 37 +++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt index 339c5b4727..f1f84ce2c4 100644 --- a/Documentation/git-bundle.txt +++ b/Documentation/git-bundle.txt @@ -44,6 +44,7 @@ header indicating what references are contained within the bundle. Like the the packed archive format itself bundles can either be self-contained, or be created using exclusions. +See the "OBJECT PREREQUISITES" section below. Bundles created using revision exclusions are "thin packs" created using the `--thin` option to linkgit:git-pack-objects[1], and @@ -152,19 +153,49 @@ contained in the union of the given bases. Each basis can be specified explicitly (e.g. `^master~10`), or implicitly (e.g. `master~10..master`, `--since=10.days.ago master`). -It is very important that the basis used be held by the destination. +OBJECT PREREQUISITES +-------------------- + +When creating bundles it is possible to create a self-contained bundle +that can be unbundled in a repository with no common history, as well +as providing negative revisions to exclude objects needed in the +earlier parts of the history. + +Feeding a revision such as `new` to `git bundle create` will create a +bundle file that contains all the objects reachable from the revision +`new`. That bundle can be unbundled in any repository to obtain a full +history that leads to the revision `new`: + +---------------- +$ git bundle create full.bundle new +---------------- + +A revision range such as `old..new` will produce a bundle file that +will require the revision `old` (and any objects reachable from it) +to exist for the bundle to be "unbundle"-able: + +---------------- +$ git bundle create full.bundle old..new +---------------- + +A self-contained bundle without any prerequisites can be extracted +into anywhere, even into an empty repository, or be cloned from +(i.e., `new`, but not `old..new`). + It is okay to err on the side of caution, causing the bundle file to contain objects already in the destination, as these are ignored when unpacking at the destination. -`git clone` can use any bundle created without negative refspecs -(e.g., `new`, but not `old..new`). If you want to match `git clone --mirror`, which would include your refs such as `refs/remotes/*`, use `--all`. If you want to provide the same set of refs that a clone directly from the source repository would get, use `--branches --tags` for the ``. +The 'git bundle verify' command can be used to check whether your +recipient repository has the required prerequisite commits for a +bundle. + EXAMPLES -------- From 0bb92f3a3a0034247868b4fcd901c18654b76c4d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Sat, 31 Jul 2021 10:23:06 +0200 Subject: [PATCH 065/762] bundle doc: elaborate on rev<->ref restriction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Elaborate on the restriction that you cannot provide a revision that doesn't resolve to a reference in the "SPECIFYING REFERENCES" section with examples. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Documentation/git-bundle.txt | 46 +++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 8 deletions(-) diff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt index f1f84ce2c4..f36939ab01 100644 --- a/Documentation/git-bundle.txt +++ b/Documentation/git-bundle.txt @@ -144,14 +144,44 @@ unbundle :: SPECIFYING REFERENCES --------------------- -'git bundle' will only package references that are shown by -'git show-ref': this includes heads, tags, and remote heads. References -such as `master~1` cannot be packaged, but are perfectly suitable for -defining the basis. More than one reference may be packaged, and more -than one basis can be specified. The objects packaged are those not -contained in the union of the given bases. Each basis can be -specified explicitly (e.g. `^master~10`), or implicitly (e.g. -`master~10..master`, `--since=10.days.ago master`). +Revisions must accompanied by reference names to be packaged in a +bundle. + +More than one reference may be packaged, and more than one basis can +be specified. The objects packaged are those not contained in the +union of the given bases. + +The 'git bundle create' command resolves the reference names for you +using the same rules as `git rev-parse --abbrev-ref=loose`. Each +basis can be specified explicitly (e.g. `^master~10`), or implicitly +(e.g. `master~10..master`, `--since=10.days.ago master`). + +All of these simple cases are OK (assuming we have a "master" and +"next" branch): + +---------------- +$ git bundle create master.bundle master +$ echo master | git bundle create master.bundle --stdin +$ git bundle create master-and-next.bundle master next +$ (echo master; echo next) | git bundle create master-and-next.bundle --stdin +---------------- + +And so are these (and the same but omitted `--stdin` examples): + +---------------- +$ git bundle create recent-master.bundle master~10..master +$ git bundle create recent-updates.bundle master~10..master next~5..next +---------------- + +A revision name or a range whose right-hand-side cannot be resolved to +a reference is not accepted: + +---------------- +$ git bundle create HEAD.bundle $(git rev-parse HEAD) +fatal: Refusing to create empty bundle. +$ git bundle create master-yesterday.bundle master~10..master~5 +fatal: Refusing to create empty bundle. +---------------- OBJECT PREREQUISITES -------------------- From 5c8273d57caf173517c9eb1ca77604709e2487b0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Sat, 31 Jul 2021 10:23:04 +0200 Subject: [PATCH 066/762] bundle doc: rewrite the "DESCRIPTION" section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrite the "DESCRIPTION" section for "git bundle" to start by talking about what bundles are in general terms, rather than diving directly into one example of what they might be used for. This changes documentation that's been substantially the same ever since the command was added in 2e0afafebd8 (Add git-bundle: move objects and references by archive, 2007-02-22). I've split up the DESCRIPTION into that section and a "BUNDLE FORMAT" section, it briefly discusses the format, but then links to the technical/bundle-format.txt documentation. The "the user must specify a basis" part of this is discussed below in "SPECIFYING REFERENCES", and will be further elaborated on in a subsequent commit. So I'm removing that part and letting the mention of "revision exclusions" suffice. There was a discussion about whether to say anything at all about "thin packs" here[1]. I think it's good to mention it for the curious reader willing to read the technical docs, but let's explicitly say that there's no "thick pack", and that the difference shouldn't matter. 1. http://lore.kernel.org/git/xmqqk0mbt5rj.fsf@gitster.g Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Documentation/git-bundle.txt | 52 +++++++++++++++++++++++++++--------- 1 file changed, 39 insertions(+), 13 deletions(-) diff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt index 53804cad4b..339c5b4727 100644 --- a/Documentation/git-bundle.txt +++ b/Documentation/git-bundle.txt @@ -18,21 +18,47 @@ SYNOPSIS DESCRIPTION ----------- -Some workflows require that one or more branches of development on one -machine be replicated on another machine, but the two machines cannot -be directly connected, and therefore the interactive Git protocols (git, -ssh, http) cannot be used. +Create, unpack, and manipulate "bundle" files. Bundles are used for +the "offline" transfer of Git objects without an active "server" +sitting on the other side of the network connection. -The 'git bundle' command packages objects and references in an archive -at the originating machine, which can then be imported into another -repository using 'git fetch', 'git pull', or 'git clone', -after moving the archive by some means (e.g., by sneakernet). +They can be used to create both incremental and full backups of a +repository, and to relay the state of the references in one repository +to another. -As no -direct connection between the repositories exists, the user must specify a -basis for the bundle that is held by the destination repository: the -bundle assumes that all objects in the basis are already in the -destination repository. +Git commands that fetch or otherwise "read" via protocols such as +`ssh://` and `https://` can also operate on bundle files. It is +possible linkgit:git-clone[1] a new repository from a bundle, to use +linkgit:git-fetch[1] to fetch from one, and to list the references +contained within it with linkgit:git-ls-remote[1]. There's no +corresponding "write" support, i.e.a 'git push' into a bundle is not +supported. + +See the "EXAMPLES" section below for examples of how to use bundles. + +BUNDLE FORMAT +------------- + +Bundles are `.pack` files (see linkgit:git-pack-objects[1]) with a +header indicating what references are contained within the bundle. + +Like the the packed archive format itself bundles can either be +self-contained, or be created using exclusions. + +Bundles created using revision exclusions are "thin packs" created +using the `--thin` option to linkgit:git-pack-objects[1], and +unbundled using the `--fix-thin` option to linkgit:git-index-pack[1]. + +There is no option to create a "thick pack" when using revision +exclusions, users should not be concerned about the difference. By +using "thin packs" bundles created using exclusions are smaller in +size. That they're "thin" under the hood is merely noted here as a +curiosity, and as a reference to other documentation + +See link:technical/bundle-format.html[the `bundle-format` +documentation] for more details and the discussion of "thin pack" in +link:technical/pack-format.html[the pack format documentation] for +further details. OPTIONS ------- From 1d9c8daef8d0c9c9f57e06eac7511c3b9355960b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Sat, 31 Jul 2021 10:23:07 +0200 Subject: [PATCH 067/762] bundle doc: replace "basis" with "prerequsite(s)" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In the preceding commits we introduced new documentation that talks about "[commit|object] prerequsite(s)", but also faithfully moved around existing documentation that talks about the "basis". Let's change both that moved-around documentation and other existing documentation in the file to consistently use "[commit|object]" prerequisite(s)" instead of talking about "basis". The mention of "basis" isn't wrong, but readers will be helped by us using only one term throughout the document for this concept. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano --- Documentation/git-bundle.txt | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt index f36939ab01..ac0d003835 100644 --- a/Documentation/git-bundle.txt +++ b/Documentation/git-bundle.txt @@ -147,13 +147,13 @@ SPECIFYING REFERENCES Revisions must accompanied by reference names to be packaged in a bundle. -More than one reference may be packaged, and more than one basis can +More than one reference may be packaged, and more than one set of prerequisite objects can be specified. The objects packaged are those not contained in the -union of the given bases. +union of the prerequisites. The 'git bundle create' command resolves the reference names for you using the same rules as `git rev-parse --abbrev-ref=loose`. Each -basis can be specified explicitly (e.g. `^master~10`), or implicitly +prerequisite can be specified explicitly (e.g. `^master~10`), or implicitly (e.g. `master~10..master`, `--since=10.days.ago master`). All of these simple cases are OK (assuming we have a "master" and @@ -236,7 +236,7 @@ but we can move data from A to B via some mechanism (CD, email, etc.). We want to update R2 with development made on the branch master in R1. To bootstrap the process, you can first create a bundle that does not have -any basis. You can use a tag to remember up to what commit you last +any prerequisites. You can use a tag to remember up to what commit you last processed, in order to make it easy to later update the other repository with an incremental bundle: @@ -287,7 +287,7 @@ machineB$ git pull If you know up to what commit the intended recipient repository should have the necessary objects, you can use that knowledge to specify the -basis, giving a cut-off point to limit the revisions and objects that go +prerequisites, giving a cut-off point to limit the revisions and objects that go in the resulting bundle. The previous example used the lastR2bundle tag for this purpose, but you can use any other options that you would give to the linkgit:git-log[1] command. Here are more examples: @@ -298,7 +298,7 @@ You can use a tag that is present in both: $ git bundle create mybundle v1.0.0..master ---------------- -You can use a basis based on time: +You can use a prerequisite based on time: ---------------- $ git bundle create mybundle --since=10.days master @@ -311,7 +311,7 @@ $ git bundle create mybundle -10 master ---------------- You can run `git-bundle verify` to see if you can extract from a bundle -that was created with a basis: +that was created with a prerequisite: ---------------- $ git bundle verify mybundle From 62a15162fe60f11c515bec116c9a1f58173a032f Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Sat, 31 Jul 2021 17:27:38 +0000 Subject: [PATCH 068/762] merge-ort: remove compile-time ability to turn off usage of memory pools Simplify code maintenance by removing the ability to toggle between usage of memory pools and direct allocations. This allows us to also remove paths_to_free since it was solely about bookkeeping to make sure we freed the necessary paths, and allows us to remove some auxiliary functions. Suggested-by: Jeff King Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- merge-ort.c | 209 ++++++++++++---------------------------------------- 1 file changed, 47 insertions(+), 162 deletions(-) diff --git a/merge-ort.c b/merge-ort.c index 86ab8f6012..88ade50f4e 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -37,8 +37,6 @@ #include "unpack-trees.h" #include "xdiff-interface.h" -#define USE_MEMORY_POOL 1 /* faster, but obscures memory leak hunting */ - /* * We have many arrays of size 3. Whenever we have such an array, the * indices refer to one of the sides of the three-way merge. This is so @@ -305,8 +303,6 @@ struct merge_options_internal { * * these keys serve to intern all the path strings, which allows * us to do pointer comparison on directory names instead of * strcmp; we just have to be careful to use the interned strings. - * (Technically paths_to_free may track some strings that were - * removed from froms paths.) * * The values of paths: * * either a pointer to a merged_info, or a conflict_info struct @@ -349,18 +345,7 @@ struct merge_options_internal { * freed together too. Using a memory pool for these provides a * nice speedup. */ - struct mem_pool internal_pool; - struct mem_pool *pool; /* NULL, or pointer to internal_pool */ - - /* - * paths_to_free: additional list of strings to free - * - * If keys are removed from "paths", they are added to paths_to_free - * to ensure they are later freed. We avoid free'ing immediately since - * other places (e.g. conflict_info.pathnames[]) may still be - * referencing these paths. - */ - struct string_list paths_to_free; + struct mem_pool pool; /* * output: special messages and conflict notices for various paths @@ -539,19 +524,7 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, void (*strset_clear_func)(struct strset *) = reinitialize ? strset_partial_clear : strset_clear; - if (opti->pool) - strmap_clear_func(&opti->paths, 0); - else { - /* - * We marked opti->paths with strdup_strings = 0, so that - * we wouldn't have to make another copy of the fullpath - * created by make_traverse_path from setup_path_info(). - * But, now that we've used it and have no other references - * to these strings, it is time to deallocate them. - */ - free_strmap_strings(&opti->paths); - strmap_clear_func(&opti->paths, 1); - } + strmap_clear_func(&opti->paths, 0); /* * All keys and values in opti->conflicted are a subset of those in @@ -560,20 +533,6 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, */ strmap_clear_func(&opti->conflicted, 0); - if (!opti->pool) { - /* - * opti->paths_to_free is similar to opti->paths; we - * created it with strdup_strings = 0 to avoid making - * _another_ copy of the fullpath but now that we've used - * it and have no other references to these strings, it is - * time to deallocate them. We do so by temporarily - * setting strdup_strings to 1. - */ - opti->paths_to_free.strdup_strings = 1; - string_list_clear(&opti->paths_to_free, 0); - opti->paths_to_free.strdup_strings = 0; - } - if (opti->attr_index.cache_nr) /* true iff opt->renormalize */ discard_index(&opti->attr_index); @@ -623,11 +582,7 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, strmap_clear(&opti->output, 0); } -#if USE_MEMORY_POOL - mem_pool_discard(&opti->internal_pool, 0); - if (!reinitialize) - opti->pool = NULL; -#endif + mem_pool_discard(&opti->pool, 0); /* Clean out callback_data as well. */ FREE_AND_NULL(renames->callback_data); @@ -693,12 +648,9 @@ static void path_msg(struct merge_options *opt, static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool, const char *path) { + /* Similar to alloc_filespec(), but allocate from pool and reuse path */ struct diff_filespec *spec; - if (!pool) - return alloc_filespec(path); - - /* Similar to alloc_filespec, but allocate from pool and reuse path */ spec = mem_pool_calloc(pool, 1, sizeof(*spec)); spec->path = (char*)path; /* spec won't modify it */ @@ -712,12 +664,9 @@ static struct diff_filepair *pool_diff_queue(struct mem_pool *pool, struct diff_filespec *one, struct diff_filespec *two) { + /* Same code as diff_queue(), except allocate from pool */ struct diff_filepair *dp; - if (!pool) - return diff_queue(queue, one, two); - - /* Same code as diff_queue, except allocate from pool */ dp = mem_pool_calloc(pool, 1, sizeof(*dp)); dp->one = one; dp->two = two; @@ -726,27 +675,6 @@ static struct diff_filepair *pool_diff_queue(struct mem_pool *pool, return dp; } -static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size) -{ - if (!pool) - return xcalloc(count, size); - return mem_pool_calloc(pool, count, size); -} - -static void *pool_alloc(struct mem_pool *pool, size_t size) -{ - if (!pool) - return xmalloc(size); - return mem_pool_alloc(pool, size); -} - -static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len) -{ - if (!pool) - return xstrndup(str, len); - return mem_pool_strndup(pool, str, len); -} - /* add a string to a strbuf, but converting "/" to "_" */ static void add_flattened_path(struct strbuf *out, const char *s) { @@ -875,9 +803,9 @@ static void setup_path_info(struct merge_options *opt, assert(!df_conflict || !resolved); /* df_conflict implies !resolved */ assert(resolved == (merged_version != NULL)); - mi = pool_calloc(opt->priv->pool, 1, - resolved ? sizeof(struct merged_info) : - sizeof(struct conflict_info)); + mi = mem_pool_calloc(&opt->priv->pool, 1, + resolved ? sizeof(struct merged_info) : + sizeof(struct conflict_info)); mi->directory_name = current_dir_name; mi->basename_offset = current_dir_name_len; mi->clean = !!resolved; @@ -924,7 +852,6 @@ static void add_pair(struct merge_options *opt, unsigned dir_rename_mask) { struct diff_filespec *one, *two; - struct mem_pool *pool = opt->priv->pool; struct rename_info *renames = &opt->priv->renames; int names_idx = is_add ? side : 0; @@ -975,11 +902,11 @@ static void add_pair(struct merge_options *opt, return; } - one = pool_alloc_filespec(pool, pathname); - two = pool_alloc_filespec(pool, pathname); + one = pool_alloc_filespec(&opt->priv->pool, pathname); + two = pool_alloc_filespec(&opt->priv->pool, pathname); fill_filespec(is_add ? two : one, &names[names_idx].oid, 1, names[names_idx].mode); - pool_diff_queue(pool, &renames->pairs[side], one, two); + pool_diff_queue(&opt->priv->pool, &renames->pairs[side], one, two); } static void collect_rename_info(struct merge_options *opt, @@ -1170,7 +1097,7 @@ static int collect_merge_info_callback(int n, len = traverse_path_len(info, p->pathlen); /* +1 in both of the following lines to include the NUL byte */ - fullpath = pool_alloc(opt->priv->pool, len + 1); + fullpath = mem_pool_alloc(&opt->priv->pool, len + 1); make_traverse_path(fullpath, len + 1, info, p->path, p->pathlen); /* @@ -1425,7 +1352,7 @@ static int handle_deferred_entries(struct merge_options *opt, copy = renames->deferred[side].possible_trivial_merges; strintmap_init_with_options(&renames->deferred[side].possible_trivial_merges, 0, - opt->priv->pool, + &opt->priv->pool, 0); strintmap_for_each_entry(©, &iter, entry) { const char *path = entry->key; @@ -2377,21 +2304,17 @@ static void apply_directory_rename_modifications(struct merge_options *opt, VERIFY_CI(ci); /* Find parent directories missing from opt->priv->paths */ - if (opt->priv->pool) { - cur_path = mem_pool_strdup(opt->priv->pool, new_path); - free((char*)new_path); - new_path = (char *)cur_path; - } else { - cur_path = new_path; - } + cur_path = mem_pool_strdup(&opt->priv->pool, new_path); + free((char*)new_path); + new_path = (char *)cur_path; while (1) { /* Find the parent directory of cur_path */ char *last_slash = strrchr(cur_path, '/'); if (last_slash) { - parent_name = pool_strndup(opt->priv->pool, - cur_path, - last_slash - cur_path); + parent_name = mem_pool_strndup(&opt->priv->pool, + cur_path, + last_slash - cur_path); } else { parent_name = opt->priv->toplevel_dir; break; @@ -2400,8 +2323,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt, /* Look it up in opt->priv->paths */ entry = strmap_get_entry(&opt->priv->paths, parent_name); if (entry) { - if (!opt->priv->pool) - free((char*)parent_name); parent_name = entry->key; /* reuse known pointer */ break; } @@ -2428,16 +2349,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt, parent_name = cur_dir; } - if (!opt->priv->pool) { - /* - * We are removing old_path from opt->priv->paths. - * old_path also will eventually need to be freed, but it - * may still be used by e.g. ci->pathnames. So, store it - * in another string-list for now. - */ - string_list_append(&opt->priv->paths_to_free, old_path); - } - assert(ci->filemask == 2 || ci->filemask == 4); assert(ci->dirmask == 0); strmap_remove(&opt->priv->paths, old_path, 0); @@ -2471,8 +2382,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt, new_ci->stages[index].mode = ci->stages[index].mode; oidcpy(&new_ci->stages[index].oid, &ci->stages[index].oid); - if (!opt->priv->pool) - free(ci); ci = new_ci; } @@ -2888,7 +2797,6 @@ static void use_cached_pairs(struct merge_options *opt, { struct hashmap_iter iter; struct strmap_entry *entry; - struct mem_pool *pool = opt->priv->pool; /* * Add to side_pairs all entries from renames->cached_pairs[side_index]. @@ -2900,30 +2808,24 @@ static void use_cached_pairs(struct merge_options *opt, const char *new_name = entry->value; if (!new_name) new_name = old_name; - if (pool) { - /* - * cached_pairs has _copies* of old_name and new_name, - * because it has to persist across merges. When - * pool != NULL - * pool_alloc_filespec() will just re-use the existing - * filenames, which will also get re-used by - * opt->priv->paths if they become renames, and then - * get freed at the end of the merge, leaving the copy - * in cached_pairs dangling. Avoid this by making a - * copy here. - * - * When pool == NULL, pool_alloc_filespec() calls - * alloc_filespec(), which makes a copy; we don't want - * to add another. - */ - old_name = mem_pool_strdup(pool, old_name); - new_name = mem_pool_strdup(pool, new_name); - } + + /* + * cached_pairs has *copies* of old_name and new_name, + * because it has to persist across merges. Since + * pool_alloc_filespec() will just re-use the existing + * filenames, which will also get re-used by + * opt->priv->paths if they become renames, and then + * get freed at the end of the merge, that would leave + * the copy in cached_pairs dangling. Avoid this by + * making a copy here. + */ + old_name = mem_pool_strdup(&opt->priv->pool, old_name); + new_name = mem_pool_strdup(&opt->priv->pool, new_name); /* We don't care about oid/mode, only filenames and status */ - one = pool_alloc_filespec(pool, old_name); - two = pool_alloc_filespec(pool, new_name); - pool_diff_queue(pool, pairs, one, two); + one = pool_alloc_filespec(&opt->priv->pool, old_name); + two = pool_alloc_filespec(&opt->priv->pool, new_name); + pool_diff_queue(&opt->priv->pool, pairs, one, two); pairs->queue[pairs->nr-1]->status = entry->value ? 'R' : 'D'; } } @@ -3031,7 +2933,7 @@ static int detect_regular_renames(struct merge_options *opt, diff_queued_diff = renames->pairs[side_index]; trace2_region_enter("diff", "diffcore_rename", opt->repo); diffcore_rename_extended(&diff_opts, - opt->priv->pool, + &opt->priv->pool, &renames->relevant_sources[side_index], &renames->dirs_removed[side_index], &renames->dir_rename_count[side_index], @@ -3082,7 +2984,7 @@ static int collect_renames(struct merge_options *opt, if (p->status != 'A' && p->status != 'R') { possibly_cache_new_pair(renames, p, side_index, NULL); - pool_diff_free_filepair(opt->priv->pool, p); + pool_diff_free_filepair(&opt->priv->pool, p); continue; } @@ -3095,7 +2997,7 @@ static int collect_renames(struct merge_options *opt, possibly_cache_new_pair(renames, p, side_index, new_path); if (p->status != 'R' && !new_path) { - pool_diff_free_filepair(opt->priv->pool, p); + pool_diff_free_filepair(&opt->priv->pool, p); continue; } @@ -3213,7 +3115,7 @@ cleanup: side_pairs = &renames->pairs[s]; for (i = 0; i < side_pairs->nr; ++i) { struct diff_filepair *p = side_pairs->queue[i]; - pool_diff_free_filepair(opt->priv->pool, p); + pool_diff_free_filepair(&opt->priv->pool, p); } } @@ -3226,7 +3128,7 @@ simple_cleanup: if (combined.nr) { int i; for (i = 0; i < combined.nr; i++) - pool_diff_free_filepair(opt->priv->pool, + pool_diff_free_filepair(&opt->priv->pool, combined.queue[i]); free(combined.queue); } @@ -3701,7 +3603,7 @@ static void process_entry(struct merge_options *opt, * the directory to remain here, so we need to move this * path to some new location. */ - new_ci = pool_calloc(opt->priv->pool, 1, sizeof(*new_ci)); + new_ci = mem_pool_calloc(&opt->priv->pool, 1, sizeof(*new_ci)); /* We don't really want new_ci->merged.result copied, but it'll * be overwritten below so it doesn't matter. We also don't @@ -3794,7 +3696,8 @@ static void process_entry(struct merge_options *opt, const char *a_path = NULL, *b_path = NULL; int rename_a = 0, rename_b = 0; - new_ci = pool_alloc(opt->priv->pool, sizeof(*new_ci)); + new_ci = mem_pool_alloc(&opt->priv->pool, + sizeof(*new_ci)); if (S_ISREG(a_mode)) rename_a = 1; @@ -3863,19 +3766,8 @@ static void process_entry(struct merge_options *opt, b_path = path; strmap_put(&opt->priv->paths, b_path, new_ci); - if (rename_a && rename_b) { + if (rename_a && rename_b) strmap_remove(&opt->priv->paths, path, 0); - /* - * We removed path from opt->priv->paths. path - * will also eventually need to be freed if not - * part of a memory pool...but it may still be - * used by e.g. ci->pathnames. So, store it in - * another string-list for now in that case. - */ - if (!opt->priv->pool) - string_list_append(&opt->priv->paths_to_free, - path); - } /* * Do special handling for b_path since process_entry() @@ -4482,13 +4374,8 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) /* Initialization of various renames fields */ renames = &opt->priv->renames; -#if USE_MEMORY_POOL - mem_pool_init(&opt->priv->internal_pool, 0); - opt->priv->pool = &opt->priv->internal_pool; -#else - opt->priv->pool = NULL; -#endif - pool = opt->priv->pool; + mem_pool_init(&opt->priv->pool, 0); + pool = &opt->priv->pool; for (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) { strintmap_init_with_options(&renames->dirs_removed[i], NOT_RELEVANT, pool, 0); @@ -4525,15 +4412,13 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) * Although we initialize opt->priv->paths with strdup_strings=0, * that's just to avoid making yet another copy of an allocated * string. Putting the entry into paths means we are taking - * ownership, so we will later free it. paths_to_free is similar. + * ownership, so we will later free it. * * In contrast, conflicted just has a subset of keys from paths, so * we don't want to free those (it'd be a duplicate free). */ strmap_init_with_options(&opt->priv->paths, pool, 0); strmap_init_with_options(&opt->priv->conflicted, pool, 0); - if (!opt->priv->pool) - string_list_init_nodup(&opt->priv->paths_to_free); /* * keys & strbufs in output will sometimes need to outlive "paths", From 3e5e6c6e94f09973d3a72049aad09fd2131a3648 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 4 Aug 2021 15:56:11 +0200 Subject: [PATCH 069/762] fetch-pack: speed up loading of refs via commit graph MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When doing reference negotiation, git-fetch-pack(1) is loading all refs from disk in order to determine which commits it has in common with the remote repository. This can be quite expensive in repositories with many references though: in a real-world repository with around 2.2 million refs, fetching a single commit by its ID takes around 44 seconds. Dominating the loading time is decompression and parsing of the objects which are referenced by commits. Given the fact that we only care about commits (or tags which can be peeled to one) in this context, there is thus an easy performance win by switching the parsing logic to make use of the commit graph in case we have one available. Like this, we avoid hitting the object database to parse these commits but instead only load them from the commit-graph. This results in a significant performance boost when executing git-fetch in said repository with 2.2 million refs: Benchmark #1: HEAD~: git fetch $remote $commit Time (mean ± σ): 44.168 s ± 0.341 s [User: 42.985 s, System: 1.106 s] Range (min … max): 43.565 s … 44.577 s 10 runs Benchmark #2: HEAD: git fetch $remote $commit Time (mean ± σ): 19.498 s ± 0.724 s [User: 18.751 s, System: 0.690 s] Range (min … max): 18.629 s … 20.454 s 10 runs Summary 'HEAD: git fetch $remote $commit' ran 2.27 ± 0.09 times faster than 'HEAD~: git fetch $remote $commit' Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- fetch-pack.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index c135635e34..89fdeb3497 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -137,8 +137,14 @@ static struct commit *deref_without_lazy_fetch(const struct object_id *oid, break; } } - if (type == OBJ_COMMIT) - return (struct commit *) parse_object(the_repository, oid); + + if (type == OBJ_COMMIT) { + struct commit *commit = lookup_commit(the_repository, oid); + if (!commit || repo_parse_commit(the_repository, commit)) + return NULL; + return commit; + } + return NULL; } From e037c2e41869489c1e17701a9f1c429ad9c18f9b Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Wed, 4 Aug 2021 23:50:45 +0000 Subject: [PATCH 070/762] git-rebase.txt: correct antiquated claims about --rebase-merges When --rebase-merges was first introduced, it only worked with the `recursive` strategy. Some time later, it gained support for merges using the `octopus` strategy. The limitation of only supporting these two strategies was documented in 25cff9f109 ("rebase -i --rebase-merges: add a section to the man page", 2018-04-25) and lifted in e145d99347 ("rebase -r: support merge strategies other than `recursive`", 2019-07-31). However, when the limitation was lifted, the documentation was not updated. Update it now. Acked-by: Johannes Schindelin Acked-by: Derrick Stolee Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- Documentation/git-rebase.txt | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt index 55af6fd24e..8a67227846 100644 --- a/Documentation/git-rebase.txt +++ b/Documentation/git-rebase.txt @@ -1219,12 +1219,16 @@ successful merge so that the user can edit the message. If a `merge` command fails for any reason other than merge conflicts (i.e. when the merge operation did not even start), it is rescheduled immediately. -At this time, the `merge` command will *always* use the `recursive` -merge strategy for regular merges, and `octopus` for octopus merges, -with no way to choose a different one. To work around -this, an `exec` command can be used to call `git merge` explicitly, -using the fact that the labels are worktree-local refs (the ref -`refs/rewritten/onto` would correspond to the label `onto`, for example). +By default, the `merge` command will use the `recursive` merge +strategy for regular merges, and `octopus` for octopus merges. One +can specify a default strategy for all merges using the `--strategy` +argument when invoking rebase, or can override specific merges in the +interactive list of commands by using an `exec` command to call `git +merge` explicitly with a `--strategy` argument. Note that when +calling `git merge` explicitly like this, you can make use of the fact +that the labels are worktree-local refs (the ref `refs/rewritten/onto` +would correspond to the label `onto`, for example) in order to refer +to the branches you want to merge. Note: the first command (`label onto`) labels the revision onto which the commits are rebased; The name `onto` is just a convention, as a nod From b378df72ed147d346503341db06771c52cd2dbb8 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Wed, 4 Aug 2021 23:50:46 +0000 Subject: [PATCH 071/762] directory-rename-detection.txt: small updates due to merge-ort optimizations In commit 0c4fd732f0 ("Move computation of dir_rename_count from merge-ort to diffcore-rename", 2021-02-27), much of the logic for computing directory renames moved into diffcore-rename. directory-rename-detection.txt had claims that all of that logic was found in merge-recursive. Update the documentation. Acked-by: Derrick Stolee Acked-by: Johannes Schindelin Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- .../technical/directory-rename-detection.txt | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/Documentation/technical/directory-rename-detection.txt b/Documentation/technical/directory-rename-detection.txt index 49b83ef3cc..029ee2cedc 100644 --- a/Documentation/technical/directory-rename-detection.txt +++ b/Documentation/technical/directory-rename-detection.txt @@ -2,9 +2,9 @@ Directory rename detection ========================== Rename detection logic in diffcore-rename that checks for renames of -individual files is aggregated and analyzed in merge-recursive for cases -where combinations of renames indicate that a full directory has been -renamed. +individual files is also aggregated there and then analyzed in either +merge-ort or merge-recursive for cases where combinations of renames +indicate that a full directory has been renamed. Scope of abilities ------------------ @@ -88,9 +88,11 @@ directory rename detection support in: Folks have requested in the past that `git diff` detect directory renames and somehow simplify its output. It is not clear whether this would be desirable or how the output should be simplified, so this was - simply not implemented. Further, to implement this, directory rename - detection logic would need to move from merge-recursive to - diffcore-rename. + simply not implemented. Also, while diffcore-rename has most of the + logic for detecting directory renames, some of the logic is still found + within merge-ort and merge-recursive. Fully supporting directory + rename detection in diffs would require copying or moving the remaining + bits of logic to the diff machinery. * am From e80178eac6f3513bed7ec7ead9d5b0ebfa429241 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Wed, 4 Aug 2021 23:50:47 +0000 Subject: [PATCH 072/762] Documentation: edit awkward references to `git merge-recursive` A few places in the documentation referred to the "`recursive` strategy" using the phrase "`git merge-recursive`", suggesting that it was forking subprocesses to call a toplevel builtin. Perhaps that was relevant to when rebase was a shell script, but it seems like a rather indirect way to refer to the `recursive` strategy. Simplify the references. Acked-by: Derrick Stolee Acked-by: Johannes Schindelin Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- Documentation/git-rebase.txt | 5 ++--- Documentation/merge-options.txt | 4 ++-- Documentation/merge-strategies.txt | 9 +++++---- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt index 8a67227846..c3edcb07e3 100644 --- a/Documentation/git-rebase.txt +++ b/Documentation/git-rebase.txt @@ -354,9 +354,8 @@ See also INCOMPATIBLE OPTIONS below. -s :: --strategy=:: - Use the given merge strategy. - If there is no `-s` option 'git merge-recursive' is used - instead. This implies --merge. + Use the given merge strategy, instead of the default + `recursive`. This implies `--merge`. + Because 'git rebase' replays each commit from the working branch on top of the branch using the given strategy, using diff --git a/Documentation/merge-options.txt b/Documentation/merge-options.txt index eb0aabd396..f819bd8dd6 100644 --- a/Documentation/merge-options.txt +++ b/Documentation/merge-options.txt @@ -112,8 +112,8 @@ With --squash, --commit is not allowed, and will fail. Use the given merge strategy; can be supplied more than once to specify them in the order they should be tried. If there is no `-s` option, a built-in list of strategies - is used instead ('git merge-recursive' when merging a single - head, 'git merge-octopus' otherwise). + is used instead (`recursive` when merging a single head, + `octopus` otherwise). -X