Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/newren/git-filter-repo/llms.txt

Use this file to discover all available pages before exploring further.

Common questions and answers about using git-filter-repo.
This is fundamental to how Git operates. In more detail…Each commit in Git is a hash of its contents. Those contents include the commit message, the author (name, email, and time authored), the committer (name, email and time committed), the toplevel tree hash, and the parent(s) of the commit. This means that if any of the commit fields change, including the tree hash or the hash of the parent(s) of the commit, then the hash for the commit will change.The same is true for files (“blobs”) and trees stored in git as well; each is a hash of its contents, so literally if anything changes, the commit hash will change.If you attempt to write a commit (or tree or blob) object with an incorrect hash, Git will reject it as corrupt.
There are two aspects to this, or two possible underlying questions users might be asking here:
  • Why did commits newer than the ones I expected have their hash change?
  • Why did commits older than the ones I expected have their hash change?
For the first question, see why filter-repo rewrites commit hashes above, and note that if you modify some old commit, perhaps to remove a file, then obviously that commit’s hash must change. Further, since that commit will have a new hash, any other commit with that commit as a parent will need to have a new hash. That will need to chain all the way to the most recent commits in history. This is fundamental to Git and there is nothing you can do to change this.For the second question, there are two causes: (1) the filter you specified applies to the older commits too, or (2) git-fast-export and git-fast-import (both of which git-filter-repo uses) canonicalize history in various ways. The second cause means that even if you have no filter, these tools sometimes change commit hashes. This can happen in any of these cases:
  • If you have signed commits, the signatures will be stripped
  • If you have commits with extended headers, the extended headers will be stripped (signed commits are actually a special case of this)
  • If you have commits in an encoding other than UTF-8, they will by default be re-encoded into UTF-8
  • If you have a commit without an author, one will be added that matches the committer
  • If you have trees that are not canonical (e.g. incorrect sorting order), they will be canonicalized
If this affects you and you really only want to rewrite newer commits in history, you can use the --refs argument to git-filter-repo to specify a range of history that you want rewritten.
git-filter-repo’s name is git-filter-repo. Obviously it is going to rewrite all branches by default.git-filter-repo can restrict its rewriting to a subset of history, such as a single branch, using the --refs option. However, using that comes with the risk that one branch now has a different version of some commits than other branches do; usually, when you rewrite history, you want all branches that depend on what you are rewriting to be updated.
Arguments to --path should be paths as Git would report them, when run from the toplevel of the git repository.Good path examples:
  • README.md
  • Documentation/README.md
  • src/modules/flux/capacitor.rs
You can find examples of valid path names from your repository by running either git diff --no-relative --name-only or git log --no-relative --name-only --format="".The following are basic rules about paths the way that Git reports and uses them:
  • do not use absolute paths
  • always treats paths as relative to the toplevel of the repository (do not add a leading slash, and do not specify paths relative to some subdirectory of the repository even if that is your current working directory)
  • do not use the special directories . or .. anywhere in your path
  • do not use \, the Windows path separator, between directories and files; always use / regardless of platform
Erroneous path examples (do NOT use any of these styles):
  • /absolute/path/to/src/modules/program.c
  • /src/modules/program.c
  • src/docs/../modules/main.java
  • scripts/config/./update.sh
  • ./tests/fixtures/image.jpg
  • ../src/main.rs
  • C:\absolute\path\to\src\modules\program.c
  • src\modules\program.c
Sure, if you followed the instructions. The instructions told you to make a fresh clone before running git-filter-repo. If you did that (and didn’t force push your rewritten history back over the original), you can just throw away your clone with the flubbed rewrite, and make a new clone.If you didn’t make a fresh clone, and you didn’t run with --force, you would have seen the following warning:
Aborting: Refusing to destructively overwrite repo history since
this does not look like a fresh clone.
[...]
Please operate on a fresh clone instead. If you want to proceed
anyway, use --force.
If you then added --force, well, you were warned.If you didn’t make a fresh clone, and you started with --force, and you didn’t think to read the description of the --force option:
Ignore fresh clone checks and rewrite history (an irreversible
operation, especially since it by default ends with an
immediate pruning of reflogs and old objects).
and you didn’t read even the beginning of the manual
git-filter-repo destructively rewrites history
and you think it’s okay to run a command with --force in it on something you don’t have a backup of, then now is the time to reassess your life choices. --force should be a pretty clear warning sign.
This will never be supported.
  • Providing an alternate method to restore would require storing both the original history and the new history, meaning that those who are trying to shrink their repository size instead see it grow and have to figure out extra steps to expunge the old history to see the actual size savings. Experience with other tools showed that this was frustrating and difficult to figure out for many users.
  • Providing an alternate method to restore would mean that users who are trying to purge sensitive data from their repository still find the sensitive data after the rewrite because it hasn’t actually been purged. In order to actually purge it, they have to take extra steps. Same as with the last bullet point, experience has shown that extra steps to purge the extra information is difficult and error-prone. This extra difficulty is particularly problematic when you’re trying to expunge sensitive data.
  • Providing an alternate method to restore would also mean trying to figure out what should be backed up and how. The obvious choices used by previous tools only actually provided partial backups (reflogs would be ignored for example, as would uncommitted changes whether staged or not). The more you try to carefully backup everything, the more difficult the restoration from backup will be. The only backup mechanism I’ve found that seems reasonable, is making a separate clone. That’s expensive to do automatically for the user (especially if the filtering is done via multiple invocations of the tool). Plus, it’s not clear where the clone should be stored, especially to avoid the previous problems for size-reduction and sensitive-data-removal folks.
  • Providing an alternate method to restore would also mean providing documentation on how to restore. Past methods by other tools in the history rewriting space suggested that it was rather difficult for users to figure out. Difficult enough, in fact, that users simply didn’t ever use them. They instead made a separate clone before rewriting history and if they didn’t like the rewrite, then they just blew it away and made a new clone to work with. Since that was observed to be the easy restoration method, I simply enforced it with this tool, requiring users who look like they might not be operating on a fresh clone to use the —force flag.
But more than all that, if there were an alternate method to restore, why would you have needed to specify the —force flag? Doesn’t its existence (and the wording of its documentation) make it pretty clear on its own that there isn’t going to be a way to restore?
Some kinds of corruption can be fixed, in conjunction with git replace. If git fsck reports warnings/errors for certain objects, you can often replace them and rewrite history.
This is impossible. If you modify commits, or the files contained in them, then you change their commit IDs; this is fundamental to Git.However, if you don’t need to modify commits, but just don’t want to download everything, then look into one of the following:
Some folks want to extract a subset of a repository, do development work on it, then bring those changes back to the original repository, and send further changes in both directions. Such a tool can be written using fast-export and fast-import, but would need to make very different design decisions than git-filter-repo did.Such a tool exists; it’s called Josh. Use it if this is your usecase.
You are probably looking for git rebase. git rebase operates on the difference between commits (“diff”), allowing you to e.g. drop or modify the diff, but then runs the risk of conflicts as it attempts to apply future diffs. If you tweak one diff in the middle, since it just applies more diffs for the remaining patches, you’ll still see your changes at the end.filter-repo, by contrast, uses fast-export and fast-import. Those tools treat every commit not as a diff but as a “use the same versions of most files from the parent commit, but make these five files have these exact contents”. Since you don’t have either the diff or ready access to the version of files from the parent commit, that makes it hard to “undo” part of the changes to some file.In short, git rebase is the tool you want for removing specific commits or otherwise operating on the diff between commits.
Sometimes two co-workers have a clone of the same repository and they run the same git-filter-repo command, and they expect to get the same new commit IDs. Often they do get the same new commit IDs, but sometimes they don’t.When people get the same commit IDs, it is only by luck; not by design. There are three reasons this is unsupported and will never be reliable:
  • Different Git versions used could cause differences in filtering - Since git fast-export and git fast-import do various canonicalizations of history, and these could change over time, having different versions of Git installed can result in differences in filtering.
  • Different git-filter-repo versions used could cause differences in filtering - Over time, git-filter-repo may include new filterings by default, or fix existing filterings, or make any other number of changes. As such, having different versions of git-filter-repo installed can result in differences in filtering.
  • Different amounts of the repository cloned or differences in local-only commits can cause differences in filtering - If the clones weren’t made at the same time, one clone may have more commits than the other. Also, both may have made local commits the other doesn’t have. These additional commits could cause history to be traversed in a different order, and filtering rules are allowed to have order-dependent rules for how they filter.
git-filter-repo is designed as a one-shot history rewriting tool. Once you have filtered one clone of the repository, you should not be using it to filter other clones. All other clones of the repository should either be discarded and recloned, or have all their history rebased on top of the rewritten history.

Build docs developers (and LLMs) love

Get started for freeTalk to us