What does git fetch really do?

Question

Now I want to merge a remote branch, naming it origin/branch1, with my local branch branch1, as my partner pushed a new commit to branch1 on remote after our last merge, and I haven't committed since his last commit and want to get update from this commit. I used the following commands:

$ git fetch origin branch1
Compressing...
*branch branch1 ->FETCH_HEAD
$ git merge origin/branch1
Already up-to-date

That was not what I intended. What was in my mind before doing this was using fetch to get what my partner added and made the remote branch origin/branch1 being updated. However, the "Already up-to-date" means I failed to get the updates in my local branch1. Then I checked the sha1 value of origin/branch1 by

$ git ls-remote origin

and found that it kept the stale value of old commit after we merged last time. It tells that the git fetch origin branch1 cannot update origin/branch1. I did another experiment, where my partner created another branch named "branch2" in his side and pushed a commit in branch2 to the remote origin. Then I still used

$ git fetch origin branch2
Compressing...
$ git merge origin/branch2
No branch named "origin/branch2"

The "Compressing" told me that the first command downloaded something in branch2 in origin successfully, however, the second command told me that there was no branch named origin/branch2! Thus I concluded that neither could git fetch origin branchname update origin/branchname locally, nor could it create a remote branch if it doesn't exist.

After I replace git fetch origin branch# with git fetch origin, all the git merges worked as I expected.

However, I often see the combination of

$ git fetch remote branch-name
$ git merge remote/branch-name

So my question is what is the difference between git fetch remote and git fetch remote branch-name? And in what conditions could I succeed in letting it work as my wish by this combination?

this is all covered in `git help fetch`, did you have a specific question that isn't addressed by the documentation? — Andrew C, Dec 11 '14 at 19:14
@AndrewC I've read the `git help fetch`, especially the part referring to `git fetch [ [...]]`, word by word. However, limited by my knowledge to git, I don't think for now it could explain the scenario I described above. Could you kindly tell exactly which part of the `help` can help me? Thanks a lot. — troore, Dec 12 '14 at 01:12
`git fetch remote branch` fetches branch to FETCH_HEAD. You normally want `git fetch remote branch:branch` which would fetch it to a local branch of the same name. When you don't specify it at all it is falling back to your config which presumably specifies that (look for a line with fetch in it in .git/config). — Andrew C, Dec 12 '14 at 01:27
@AndrewC Thanks, Andrew. I know that. But what I am wondering is if `git fetch origin branch1` would update the remote counterpart locally, which is `origin/branch1`. I thought it would, but it didn't at my side. I have to use `git fetch origin` all the time to update all remote branches when I want to update only one dedicated remote branch. — troore, Dec 12 '14 at 07:37
You need to post the section of your config for the remote, specifically the fetch specifiers. — Andrew C, Dec 12 '14 at 16:58
@AndrewC My config for fetch is: `fetch = +refs/heads/*:refs/remotes/origin/*`. Thanks to your tips and the help of `git fetch`, I understand the usage of `git fetch remote branch:branch`. But I still don't understand the meaning of the behavior of "falling back" when I don't specify a local branch. — troore, Dec 13 '14 at 05:46
I think the behavior changed to address that so 'git fetch origin branch' works to update origin/branch in addition to FETCH_HEAD. I'd upgrade to a newer version — Andrew C, Dec 14 '14 at 05:07

kenorb · Answer 1 · 2016-03-27T02:57:42.413

If you need to know what exactly git fetch is doing or any other git command, just prefix it with GIT_TRACE=1, so it'll give you trace output with any other commands which are invoked, e.g.

$ GIT_TRACE=1 git fetch origin master
03:08:15.704945 git.c:348               trace: built-in: git 'fetch' 'origin' 'master'
03:08:15.706183 run-command.c:347       trace: run_command: 'ssh' 'git@github.com' 'git-upload-pack '\''FOO/BAR.git'\'''
03:08:16.006394 run-command.c:347       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
03:08:16.013096 run-command.c:347       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all'
03:08:16.013625 exec_cmd.c:129          trace: exec: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all'
03:08:16.016617 git.c:348               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all'
From github.com:FOO/BAR
 * branch            master     -> FETCH_HEAD
03:08:16.153070 run-command.c:347       trace: run_command: 'gc' '--auto'
03:08:16.153748 exec_cmd.c:129          trace: exec: 'git' 'gc' '--auto'
03:08:16.157704 git.c:348               trace: built-in: git 'gc' '--auto'

which basically is doing ssh to the remote host, running git-upload-pack which send objects packed back to git-fetch-pack which receives missing objects from another repository.

In man git-upload-pack we can read:

Invoked by git fetch-pack, learns what objects the other side is missing, and sends them after packing.

And in man git-fetch-pack we can read:

Invokes git-upload-pack on a possibly remote repository and asks it to send objects missing from this repository, to update the named heads. The list of commits available locally is found out by scanning the local refs/ hierarchy and sent to git-upload-pack running on the other end.

To answer the question, the difference between git fetch remote and git fetch remote branch-name, is that when you don't specify <refspec> parameter (such as branch), it fetches all branches and/or tags (refs, see: git ls-refs) from one or more other repositories along with the objects necessary to complete their histories. By default only tags which are reachable by the objects that are fetched are downloaded (e.g. you end up only with tags that point at branches that you are interested in).

And when you run with explicit branches and/or tags, git first determines what needs to be fetched, then getting only the relevant refs (branches or tags). E.g. git fetch origin master will fetch only the master branch.

VonC · Answer 2 · 2021-04-10T01:19:42.297

running git-upload-pack which send objects packed back to git-fetch-pack which receives missing objects from another repository.

You can see more about git fetch-pack with Git 2.19 (Q3 2018), since "git fetch-pack --all" used to unnecessarily fail upon seeing an annotated tag that points at an object other than a commit.

See commit c12c9df (13 Jun 2018) by Kirill Smelkov (navytux).
Helped-by: Junio C Hamano (gitster).
See commit e9502c0 (11 Jun 2018) by Jeff King (peff).
Helped-by: Junio C Hamano (gitster).
^{(Merged by Junio C Hamano -- gitster -- in commit 0079732, 28 Jun 2018)}

Fetch-pack --all became broken with respect to unusual tags in 5f0fc64 (fetch-pack: eliminate spurious error messages, 2012-09-09, Git v1.8.0), and was fixed only recently in e9502c0 (fetch-pack: don't try to fetch peel values with --all, 2018-06-11).

fetch-pack: don't try to fetch peel values with --all

When "fetch-pack --all" sees a tag-to-blob on the remote, it tries to fetch both the tag itself ("refs/tags/foo") and the peeled value that the remote advertises ("refs/tags/foo^{}").
Asking for the object pointed to by the latter can cause upload-pack to complain with "not our ref", since it does not mark the peeled objects with the OUR_REF (unless they were at the tip of some other ref).

Arguably upload-pack should be marking those peeled objects. But it never has in the past, since clients would generally just ask for the tag and expect to get the peeled value along with it.
And that's how "git fetch" works, as well as older versions of "fetch-pack --all".

Let's explicitly test all relevant cases with 4 tag objects pointing to

a blob,

a tree,

a commit, and

another tag objects.

The referenced tag objects themselves are referenced from under regular refs/tags/* namespace.

Before e9502c0 (Git 2.19) fetch-pack --all was failing e.g. this way:
   .../git/t/trash directory.t5500-fetch-pack/fetchall$ git ls-remote ..
   44085874...        HEAD
   ...
   bc4e9e1f...        refs/tags/tag-to-blob
   038f48ad...        refs/tags/tag-to-blob^{}    # peeled
   520db1f5...        refs/tags/tag-to-tree
   7395c100...        refs/tags/tag-to-tree^{}    # peeled

   .../git/t/trash directory.t5500-fetch-pack/fetchall$ git fetch-pack --all ..
   fatal: A git upload-pack: not our ref 038f48ad...
   fatal: The remote end hung up unexpectedly

With Git 2.25 (Q1 2020), fetch-pack includes trace2 annotation.

See commit 9e5afdf (19 Nov 2019) by Erik Chen (erikchen).
^{(Merged by Junio C Hamano -- gitster -- in commit 76c6824, 05 Dec 2019)}

fetch: add trace2 instrumentation

^{Signed-off-by: Erik Chen}

Add trace2 regions to fetch-pack.c to better track time spent in the various phases of a fetch:

parsing remote refs and finding a cutoff

marking local refs as complete

marking complete remote refs as common

All stages could potentially be slow for repositories with many refs.

Example: (you can then follow what a git fetch does)

D:\git\git>set GIT_TRACE2_EVENT=1

D:\git\git>git fetch
23:58:43.225088 exec-cmd.c:237          trace: resolved executable dir: D:/newenv/prgs/gits/PortableGit-2.24.0.2-64-bit.7z/mingw64/bin

{"event":"version","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.234089Z","file":"common-main.c","line":48,"evt":"2","exe":"2.24.0.windows.2"}
{"event":"start","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.234089Z","file":"common-main.c","line":49,"t_abs":0.011593,"argv":["git.exe","fetch"]}
{"event":"data_json","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.260090Z","file":"compat/win32/trace2_win32_process_info.c","line":118,"repo":0,"t_abs":0.037686,"t_rel":0.037686,"nesting":1,"category":"process","key":"windows/ancestry","value":["git.exe","cmd.exe","explorer.exe"]}
{"event":"def_repo","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.261089Z","file":"repository.c","line":130,"repo":1,"worktree":"D:/git/git"}

23:58:43.262092 git.c:439               trace: built-in: git fetch
{"event":"cmd_name","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.262092Z","file":"git.c","line":440,"name":"fetch","hierarchy":"fetch"}
{"event":"region_enter","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.281090Z","file":"builtin/fetch.c","line":1361,"repo":1,"nesting":1,"category":"fetch","label":"remote_refs"}
{"event":"child_start","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.284088Z","file":"run-command.c","line":735,"child_id":0,"child_class":"remote-https","use_shell":false,"argv":["git","remote-https","origin","https://github.com/git/git"]}
23:58:43.284088 run-command.c:663       trace: run_command: GIT_DIR=.git git remote-https origin https://github.com/git/git

With Git 2.31 (Q1 2021), the ls-refs protocol operation (used also by git fetch) has been optimized to narrow the sub-hierarchy of refs/ it walks to produce response.

See commit b3970c7, commit 16b1985 (20 Jan 2021) by Taylor Blau (ttaylorr).
See commit 83befd3 (20 Jan 2021) by Jacob Vosmaer (jacobvosmaer).
^{(Merged by Junio C Hamano -- gitster -- in commit 6254fa1, 05 Feb 2021)}

ls-refs.c: traverse prefixes of disjoint "ref-prefix" sets

^{Original-patch-by: Jacob Vosmaer}
^{Signed-off-by: Taylor Blau}

ls-refs performs a single revision walk over the whole ref namespace, and sends ones that match with one of the given ref prefixes down to the user.

This can be expensive if there are many refs overall, but the portion of them covered by the given prefixes is small by comparison.

To attempt to reduce the difference between the number of refs traversed, and the number of refs sent, only traverse references which are in the longest common prefix of the given prefixes.
This is very reminiscent of the approach taken in b31e268 ("ref-filter.c: find disjoint pattern prefixes", 2019-06-26, Git v2.23.0-rc0 -- merge listed in batch #6) which does an analogous thing for multi-patterned 'git for-each-ref'^(man) invocations.

The callback 'send_ref' is resilient to ignore extra patterns by discarding any arguments which do not begin with at least one of the specified prefixes.

Similarly, the code introduced in b31e268 is resilient to stop early at metacharacters, but we only pass strict prefixes here.
At worst we would return too many results, but the double checking done by send_ref will throw away anything that doesn't start with something in the prefix list.

Finally, if no prefixes were provided, then implicitly add the empty string (which will match all references) since this matches the existing behavior (see the "no restrictions" comment in "ls-refs.c:ref_match()").

What does git fetch really do?

2 Answers2

`fetch-pack`: don't try to fetch peel values with `--all`

`fetch`: add `trace2` instrumentation

`ls-refs.c`: traverse prefixes of disjoint "ref-prefix" sets

Linked

What does git fetch really do?

2 Answers2

fetch-pack: don't try to fetch peel values with --all

fetch: add trace2 instrumentation

ls-refs.c: traverse prefixes of disjoint "ref-prefix" sets

Linked

`fetch-pack`: don't try to fetch peel values with `--all`

`fetch`: add `trace2` instrumentation

`ls-refs.c`: traverse prefixes of disjoint "ref-prefix" sets