35

I have a clone. I want to reduce the history on it, without cloning from scratch with a reduced depth. Worked example:

$ git clone git@github.com:apache/spark.git
# ...
$ cd spark/
$ du -hs .git
193M    .git

OK, so that's not so but, but it'll serve for this discussion. If I try gc it gets smaller:

$ git gc --aggressive
Counting objects: 380616, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (278136/278136), done.
Writing objects: 100% (380616/380616), done.
Total 380616 (delta 182748), reused 192702 (delta 0)
Checking connectivity: 380616, done.
$ du -hs .git
108M    .git

Still, pretty big though (git pull suggests that it's still push/pullable to the remote). How about repack?

$ git repack -a -d --depth=5
Counting objects: 380616, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (95388/95388), done.
Writing objects: 100% (380616/380616), done.
Total 380616 (delta 182748), reused 380616 (delta 182748)
Pauls-MBA:spark paul$ du -hs .git
108M    .git

Yup, didn't get any smaller. --depth for repack isn't the same for clone:

$ git clone --depth 1 git@github.com:apache/spark.git
Cloning into 'spark'...
remote: Counting objects: 8520, done.
remote: Compressing objects: 100% (6611/6611), done.
remote: Total 8520 (delta 1448), reused 5101 (delta 710), pack-reused 0
Receiving objects: 100% (8520/8520), 14.82 MiB | 3.63 MiB/s, done.
Resolving deltas: 100% (1448/1448), done.
Checking connectivity... done.
Checking out files: 100% (13386/13386), done.
$ cd spark
$ du -hs .git
17M .git

Git pull says it's still in step with the remote, which surprises nobody.

OK - so how to change an existing clone to a shallow clone, without nixing it and checking it out afresh?

paul_h
  • 1,591
  • 2
  • 16
  • 27
  • What do you wish to do? you wish to work on multiple branches simultaneously? this is why you do the re-clone? – CodeWizard Jul 03 '16 at 17:27
  • What's wrong with cloning it again? That said, see [Section 7.13](https://git-scm.com/book/en/v2/Git-Tools-Replace) of the Pro Git book. It walks you through splitting a repository into two, one with recent commits only, the other retaining historical data. – chepner Jul 03 '16 at 20:26
  • @CodeWizard I've filled my SSD drive, and want some space back, without deleting whole clones. – paul_h Jul 05 '16 at 00:25

4 Answers4

36

since at least git version 2.14.1 (september 2017) there is

git fetch --depth 10

this will fetch the newest commits from origin (if there are any) and then cut off the local history to depth of 10 (if it was longer).

for normal purposes your git history is now at length of 10. but beware that the old commits still linger in your local repository and that they still exist in the remote repository.

if your aim was to have a shorter log because you currently don't need years worth of commit history then you are done. your log will be short and most common git commands now only see 10 commits.

if your aim was to free disk space because older commits have huge binary blobs which you don't need to work now then you have to actually remove the old commits from your local repository. see below for a short description how to do so.

if your aim was to actually remove the old commits (for example to remove a password from old commits) then you need to remove the commits from the remote repository. also from all clones of the remote repository. see below for links with more info on how to remove commits from remote repo.

how to remove the old commits to free disk space.

data loss warning! read the notes and pay attention to what you are doing.

in short: to actually remove the commits to free the disk space you need to remove all references that are holding them. that is (as far as i know) the reflog and the tags. also branches and stashes.

to clear the reflog:

git reflog expire --expire=all --all

to remove all tags:

git tag -l | xargs git tag -d

branches are a bit more complicated than tags. think for yourself how to handle your branches.

as for stashes; they should be temporary anyways. so just drop them like it's hot.

git stash drop

once you have removed all references you can call git garbage collector to remove dangling commits:

git gc --prune=all

now the old commits should be completely removed from disk.

note about the remove all tags command: the command will remove all tags from your local repository. if all your tags are also on the remote then this is fine. the next git fetch will refetch the relevant tags. but if you have tags which are only in your local repository then you need to backup them somehow.

the reflog is cleared automatically after certain time (90 days?) by automatic git gc. tags however will stay around forever. so if you want to free disk space from old commits you have to at least remove the tags manually.

the reflog is something like a local history of past local repository states. many git commands will record the previous state of the local repository in the reflog. with the reflog you can undo some commands or at least retrieve lost data if you made a mistake. so think before you clear the reflog.

the reflog is entirely local to your local repository.

see also

https://linuxhint.com/git-shallow-clone-and-clone-depth/

http://gitready.com/intermediate/2009/02/09/reflog-your-safety-net.html

How do I edit past git commits to remove my password from the commit logs?

Delete all local git branches

Lesmana
  • 24,114
  • 8
  • 78
  • 84
  • 1
    `git fetch --depth X` allows not only reduce but also increase depth of a repository – ephemerr Oct 08 '18 at 11:04
  • 1
    This works perfectly. To restore to full history, use `git fetch --unshallow`. – iwat Dec 12 '18 at 18:27
  • I got here from https://stackoverflow.com/questions/4698759/converting-git-repository-to-shallow/40452701 which is basically the same question stated more concisely. I use the reflog for recovering from mistakes a lot, and am surprised to see it recommended to remove it when it clears itself anyway. Is it important to remove? I feel like people should at least be given a warning of its importance. – fuzzyTew Aug 16 '20 at 15:54
  • it is only important to clear the reflog if you want the old commits to be removed now. which is presumably the case if you want to shallow your git repo. i mentioned it here because people will likely be confused why the disk space is still occupied after shallowing the repo. if you have a link to an article highlighting the importance of the reflog then please post it here. i will include it in my answer. – Lesmana Aug 16 '20 at 17:29
  • Here's a link from google: http://gitready.com/intermediate/2009/02/09/reflog-your-safety-net.html . Usually the reflog expires after 90 days which is pretty long ... but shouldn't use that much space unless you're storing big binary files in your repo or something. – fuzzyTew Aug 30 '20 at 18:22
19
git clone --mirror --depth=5  file://$PWD ../temp
rm -rf .git/objects
mv ../temp/{shallow,objects} .git
rm -rf ../temp

This really isn't cloning "from scratch", as it's purely local work and it creates virtually nothing more than the shallowed-out pack files, probably in the tens of kbytes total. I'd venture you're not going to get more efficient than this, you'll wind up with custom work that uses more space in the form of scripts and test work than this does in the form of a few kb of temporary repo overhead.

jthill
  • 48,781
  • 4
  • 72
  • 120
  • 1
    I like this (and upvoted it), but it does seem a bit "chummy with the implementation", as DMR once said about something else. In particular it assumes quite a bit about the object storage and the `shallow` file, both of which are implementation details. – torek Jul 09 '16 at 03:01
  • I like or just trust the design boundaries in git and the way they support its "[full access to internals](https://www.kernel.org/pub/software/scm/git/docs/#_description)". If it's that prominent in the main description, I'll rely on it. I'd be leerier about the _contents_ of a second-order feature like shallow, but however git does its bookkeeping for that there's no reason to avoid and still less change keeping it in a file named "shallow". @torek – jthill Jul 09 '16 at 03:52
  • With his method I'm left with a lot of broken refs. It doesn't really break anything, but it seems less than ideal. I "fixed" it for me by doing a bare clone from the remote. – Jochem Fuchs Nov 17 '16 at 11:02
  • Correction, that also failed. I ended up doing a new clean clone anyway. Still I'd like to know how to prevent this. As the suggested method seems less than ideal – Jochem Fuchs Nov 17 '16 at 12:31
  • I neglected to think about the remote-tracking refs, sorry. Adding `--mirror` will take care of that. – jthill Nov 17 '16 at 15:36
  • I could fix the 'does not point to a valid object!' problems by issuing these two commands before running the code in the answer: `git tag -l | xargs git tag -d`, `git branch -rl --format '%(refname)' | sed 's|refs/remotes/||g' | xargs git branch -rd` – vsz Aug 14 '19 at 00:10
3

Edit, Feb 2017: this answer is now outdated / wrong. Git can make a shallow clone shallower, at least internally. Git 2.11 also has --deepen to increase the depth of a clone, and it looks as though there are eventual plans to allow negative values (though right now they are rejected). It's not clear how well this works in the real world, and your best bet is still to clone the clone, as in jthill's answer.

You can only deepen a repository. This is primarily because Git is built around adding new stuff. The way shallow clones work is that your (receiving) Git gets the sender (another Git) to stop sending "new stuff" upon reaching the shallow-clone-depth argument, and coordinates with the sender so as to understand why they have stopped at that point even though more history is obviously required. They then write the IDs of "truncated" commits into a special file, .git/shallow, that both marks the repository as shallow, and notes which commits are truncated.

Note that during this process, your Git is still adding new stuff. (Also, when it has finished cloning and exits, Git forgets what the depth was, and over time it becomes impossible even to figure out what it was. All Git can tell is that this is a shallow clone, because the .git/shallow file containing commit IDs still exists.)

The rest of Git continues to be built around this "add new stuff" concept, so you can deepen the clone, but not increase its shallowness. (There's no good, agreed-upon verb for this: the opposite of deepening a pit is filling it in, but fill has the wrong connotation. Diminish might work; I think I'll use that.)

In theory, git gc, which is the only part of Git that ever actually throws anything out,1 could perhaps diminish a repository, even converting a full clone into a shallow one, but no one has written code to do that. There are some tricky bits, e.g., do you discard tags? Shallow clones start out sans tags for implementation reasons, so converting a repository to shallow, or diminishing an existing shallow repository, might call for discarding at least some tags. Certainly any tag pointing to a commit wiped out by the diminish action would have to go.

Meanwhile, the --depth argument to git-pack-objects (passed through from git repack) means something else entirely: it's the maximum length of a delta chain, when Git uses its modified xdelta compression on Git objects stored in each pack-file. This has nothing to do with the depth of particular parts of the commit DAG (as computed from each branch head).

1Well, git repack winds up throwing things out as a side effect, depending on which flags are used, but it's invoked this way by git gc. This is also true of git prune. For these two commands to really do their job properly, they need git reflog expire run first. The "normal user" end of the clean-things-up sequence is git gc; it deals with all of this. So we can say that git gc is how you discard accumulated "new stuff" that turned out to be unwanted after all.

Community
  • 1
  • 1
torek
  • 389,216
  • 48
  • 524
  • 664
0

OK here's an attempt to bash it, that ignores non-default branches, and also assumed the remote is called 'origin':

#!/bin/sh

set -e

mkdir .git_slimmer

cd $1

changed_lines=$(git status --porcelain | wc -l)
ahead_of_remote=$(git status | grep "Your branch is ahead" | wc -l)
remote_url=$(git remote show origin  | grep Fetch | cut -d' ' -f5)
latest_sha=$(git log | head -n 1 | cut -d' ' -f2)

cd ..

if [ "$changed_lines" -gt "0" ]
then
  echo "Untracked Changes - won't make the clone slimmer in that situation"
  exit 1
fi

if [ "$ahead_of_remote" -gt "0" ]
then
  echo "Local commits not in the remote - won't make the clone slimmer in that situation"
  exit 1
fi

cd .git_slimmer
git clone $remote_url --no-checkout --depth 1 foo
cd foo
latest_sha_for_new=$(git log | head -n 1 | cut -d' ' -f2)
cd ../..

if [ "$latest_sha" == "$latest_sha_for_new" ]
then
  mv "$1/.git" "$1/.gitOLD"
  mv ".git_slimmer/foo/.git" "$1/"
  rm -rf "$1/.gitOLD"
  cd "$1"
  git add .
  cd ..
else
  echo "SHA from head of existing get clone does not match the latest one from the remote: do a git pull first"
  exit 1
fi

rm -rf .git_slimmer

Use: 'git-slimmer.sh <folder_containing_git_repo>'

paul_h
  • 1,591
  • 2
  • 16
  • 27