3 tools · 2 agents · 6 scenarios · k=10 · 2026-07-20 · provenance ↓

A version-control benchmark for coding agents

Which version-control tool should you give your coding agent? Claude Code and Codex run the same version-control tasks three ways: with plain git, with Jujutsu, and with GitButler. Only the tool changes.

The fresh six-scenario matrix passed 359 of 360 runs. GitButler passed all 120 and ran about 65% faster than git with roughly 78% fewer commands; Jujutsu had the lone miss and ran slower than git overall.

Results

Results matrix

Agent

Scenario	git				Jujutsu				GitButler
Scenario	pass	time	cmds	KB	pass	time	cmds	KB	pass	time	cmds	KB
	10/10all runs passed, 95% CI 72–100%	61.3s	17.8	14.4	10/10all runs passed, 95% CI 72–100%	94.3s	17.1	32.3	10/10all runs passed, 95% CI 72–100%	20.7s	2.0	7.6
	10/10all runs passed, 95% CI 72–100%	169s	35.7	77.6	10/10all runs passed, 95% CI 72–100%	124s	22.4	71.4	10/10all runs passed, 95% CI 72–100%	29.0s	5.0	14.2
	10/10all runs passed, 95% CI 72–100%	118s	30.0	25.4	10/10all runs passed, 95% CI 72–100%	188s	40.1	89.6	10/10all runs passed, 95% CI 72–100%	35.8s	6.0	20.6
	10/10all runs passed, 95% CI 72–100%	50.5s	9.1	8.0	10/10all runs passed, 95% CI 72–100%	53.1s	9.9	13.8	10/10all runs passed, 95% CI 72–100%	21.1s	2.0	5.9
	10/10all runs passed, 95% CI 72–100%	44.4s	12.7	7.1	10/10all runs passed, 95% CI 72–100%	51.9s	10.0	18.8	10/10all runs passed, 95% CI 72–100%	23.2s	3.0	6.7
	10/10all runs passed, 95% CI 72–100%	111s	16.9	58.0	10/10all runs passed, 95% CI 72–100%	148s	26.2	50.8	10/10all runs passed, 95% CI 72–100%	74.4s	9.1	24.5
All scenarios	60/60all runs passed, 95% CI 94–100%	92.2s	20.4	31.7	60/60all runs passed, 95% CI 94–100%	110s	21.0	46.1	60/60all runs passed, 95% CI 94–100%	34.0s	4.5	13.3

	pass	time	cmds	KB
git	10/10all runs passed, 95% CI 72–100%	61.3s	17.8	14.4
Jujutsu	10/10all runs passed, 95% CI 72–100%	94.3s	17.1	32.3
GitButler	10/10all runs passed, 95% CI 72–100%	20.7s	2.0	7.6

	pass	time	cmds	KB
git	10/10all runs passed, 95% CI 72–100%	169s	35.7	77.6
Jujutsu	10/10all runs passed, 95% CI 72–100%	124s	22.4	71.4
GitButler	10/10all runs passed, 95% CI 72–100%	29.0s	5.0	14.2

	pass	time	cmds	KB
git	10/10all runs passed, 95% CI 72–100%	118s	30.0	25.4
Jujutsu	10/10all runs passed, 95% CI 72–100%	188s	40.1	89.6
GitButler	10/10all runs passed, 95% CI 72–100%	35.8s	6.0	20.6

	pass	time	cmds	KB
git	10/10all runs passed, 95% CI 72–100%	50.5s	9.1	8.0
Jujutsu	10/10all runs passed, 95% CI 72–100%	53.1s	9.9	13.8
GitButler	10/10all runs passed, 95% CI 72–100%	21.1s	2.0	5.9

	pass	time	cmds	KB
git	10/10all runs passed, 95% CI 72–100%	44.4s	12.7	7.1
Jujutsu	10/10all runs passed, 95% CI 72–100%	51.9s	10.0	18.8
GitButler	10/10all runs passed, 95% CI 72–100%	23.2s	3.0	6.7

	pass	time	cmds	KB
git	10/10all runs passed, 95% CI 72–100%	111s	16.9	58.0
Jujutsu	10/10all runs passed, 95% CI 72–100%	148s	26.2	50.8
GitButler	10/10all runs passed, 95% CI 72–100%	74.4s	9.1	24.5

All scenarios

	pass	time	cmds	KB
git	60/60all runs passed, 95% CI 94–100%	92.2s	20.4	31.7
Jujutsu	60/60all runs passed, 95% CI 94–100%	110s	21.0	46.1
GitButler	60/60all runs passed, 95% CI 94–100%	34.0s	4.5	13.3

5/5all runs passed pass rate: a wrong history fails, no matter how fast
bold = best of the three tools
click a scenario to see its per-run times
hover a pass chip for its Wilson 95% interval
KB is comparable within one agent only

Statistical read: paired per-scenario deltas vs git (95% CI)

Scenarios where every run passed (the reliability gate): git 6/6 · Jujutsu 6/6 · GitButler 6/6

Jujutsu: wall +17.6s [−23.3s, +58.6s] · commands +0.6 [−8.5, +9.6] · pass rate 0pp [0pp, 0pp] over 6 scenarios
GitButler: wall −58.2s [−106s, −10.7s] · commands −15.8 [−26.0, −5.7] · pass rate 0pp [0pp, 0pp] over 6 scenarios

Each scenario contributes one paired difference; intervals are t-based with df = scenarios − 1. An interval crossing zero means the effect is not established on this scenario set.

Codex: Both agents run the same matrix to check the tool effect holds across models, not a Claude-versus-Codex comparison.

Scenarios

Each scenario is a pre-built Git repository plus a plain-English instruction. No code is written during a run; only the version-control operation is measured.

Selective commit from a mixed working tree

A dirty working tree mixes an input-validation fix with unrelated logging, config, and debug-note edits. The task is a single commit on a new branch holding only the validation work, which spans three files, with everything else left uncommitted.

The crux

One file, src/handler.ts, has changes on both sides: two validation hunks belong in the commit while a logging hunk must stay behind. Picking the right hunks without sweeping in the rest is the whole task.

Instruction given to the agent

Commit just the input validation work on a new branch. Leave the logging/config cleanup and debug notes uncommitted.

tasks/pilot-1-selective-validation ↗

amber hunks → one commit · hollow stays uncommitted

Amend fixes into multiple earlier commits

A five-commit branch has uncommitted changes in the working tree that belong to three of its commits. The task routes each change into its matching commit by amending, not by adding a new one, and leaves a config tweak, a debug helper, and notes uncommitted.

The crux

The changes are hunks spread across shared files: src/lead.ts, the tests, and README each feed more than one target commit, so a single file must be split across different amends.

Instruction given to the agent

Amend the existing five-commit `amend-series` branch. Do not create a new commit. Route the already-present dirty changes like this: - Amend the validation helper changes in `src/lead.ts`, the malformed-email test in `tests/lead.test.ts`, and the validation wording in `README.md` into commit `refactor validation helpers`. - Amend the scoring changes in `src/lead.ts` and the enterprise-domain scoring test in `tests/lead.test.ts` into commit `add lead scoring`. - Amend the response-behavior documentation changes in `README.md` and `docs/response.md` into commit `document response behavior`. Leave the config logging change, the debug lead summary helper, and the investigation notes uncommitted.

tasks/pilot-2-multi-amend ↗

each fix amended into its commit
↳ debug · config notes stay dirty

Split a non-top commit

A commit in the middle of the branch mixes validation, scoring, and documentation changes, plus stray debug edits, and a later commit is built on top of it. The instruction asks for that commit to be split into three ordered single-purpose commits, with the debug edits returned to the working tree as uncommitted changes and the commit above left in place.

The crux

The commit is not the most recent one: rewriting it requires rebuilding every commit above it without changing their contents.

Instruction given to the agent

Split the non-top commit `add lead workflow` on the existing `split-workflow` branch. Do not keep the original broad commit. Replace it with these three commits, in this order, below the existing top commit `add handler routing metadata`: - `refactor validation helpers`: the validation helper changes in `src/lead.ts` and the malformed-email test in `tests/lead.test.ts`. - `tune lead scoring`: the enterprise-domain scoring changes in `src/lead.ts` and the enterprise-domain scoring test in `tests/lead.test.ts`. - `document lead workflow`: the workflow documentation changes in `README.md` and `docs/lead-workflow.md`. Keep `add handler routing metadata` as the top commit after the split. Leave the config logging change, the debug lead summary helper, and the investigation notes uncommitted.

tasks/pilot-3-split-commit ↗

one commit split into three, top kept
↳ stays uncommitted: debug · config notes

Reorder a block of commits

The branch's contents are correct, but the retry and notification commits appear after commits that logically depend on them. The instruction asks for that block to be moved earlier in the history, with every commit's contents and message unchanged and nothing left uncommitted.

The crux

The reordering must preserve each commit's contents and message exactly; an incorrect sequence of moves produces conflicts.

Instruction given to the agent

Reorder the existing commits on the `reorder-series` branch. Move the adjacent delivery-related block (`add retry policy` and `add notification sender`) earlier in the branch. Do not change any file contents and do not create functional changes. Final commit order must be exactly this, oldest to newest: 1. `add app configuration` 2. `add retry policy` 3. `add notification sender` 4. `add customer model` 5. `add email formatter` 6. `document notification flow` The commit messages and each commit's content should stay attached to the same subject. Leave the worktree clean.

tasks/pilot-4-reorder-commits ↗

delivery block moved earlier · same contents

Squash commit groups

A seven-commit branch interleaves two standalone commits with two runs of incremental commits: a two-commit parser group and a three-commit retry group. The task squashes each run into one semantic commit and leaves the standalone commits untouched, ending with four.

The crux

Only the two adjacent groups may be combined; the standalone commits must stay separate, and the final file contents must be identical with a clean worktree.

Instruction given to the agent

Squash commit groups on the existing `squash-series` branch. Do not change any file contents and do not create functional changes. Keep these commits as separate commits: - `add parser token model` - `add export endpoint` Squash these adjacent commit groups: - Squash `extract parser helpers` and `wire parser helpers` into one commit named `add parser pipeline`. - Squash `add retry option`, `test retry option`, and `document retry option` into one commit named `add retry support`. The final branch order, oldest to newest, should be: 1. `add parser token model` 2. `add parser pipeline` 3. `add export endpoint` 4. `add retry support` Leave the worktree clean.

tasks/pilot-5-squash-commits ↗

two groups squashed · two commits kept

Update a dirty branch onto a moved target

Main advanced by two commits while a feature branch accumulated two commits plus uncommitted work. The task rebuilds the feature branch on the new main tip, resolves both commit conflicts, and carries the dirty worktree through unchanged.

The crux

The update must keep linear history and resolve two conflicts differently while preserving a tracked edit and an untracked note as exact uncommitted leftovers.

Instruction given to the agent

The `main` branch has moved ahead while work on `notify-retry` was in progress. Bring `notify-retry` up to date: rebuild it on top of the current `main` with linear history, keeping its two commits and their messages intact. Do not modify `main` itself. Two incoming changes clash with this branch's work: - `src/notify.ts`: keep both notification channels — the `sms` channel added on this branch and the `push` channel added on `main`. - `src/config.ts`: this branch deliberately tuned the retry limit to 4; keep that over the value `main` bumped it to. The worktree also has uncommitted work in progress: a README edit and an untracked rollout checklist. Both must survive the update exactly as they are and stay uncommitted. Leave no conflict markers, unmerged files, or half-finished operations behind.

tasks/pilot-6-update-dirty-branch ↗

rebuilt on new main · conflicts resolved
↳ stays uncommitted: README edit · rollout note

Method

A hidden, deterministic grader scores the final Git state, so two command sequences pass if they produce the same history. Every tool gets the same task and plain-English instruction, its name never appears in the prompt, and setup is excluded from timing.

Identical instruction across tools: Each task is one prepared repository plus one plain-English instruction ("commit just the input-validation work on a new branch, leave the rest uncommitted"). The tool’s name never appears; the agent decides how to carry it out.
Deterministic grader: A hidden, scripted check that returns the same verdict for the same final state, with no LLM judge. It inspects commit boundaries, branch topology, and what stayed uncommitted; it never compares commands against a reference.
Timing boundary: Fixture build, workspace prep, skill install, and placing the uncommitted changes all happen before timing starts. The figures cover only the agent’s work on the task.
Git write restriction: In GitButler and Jujutsu runs, raw git write commands are blocked, so the agent must use the tool under test. Git the tool calls internally is the tool’s own work, not the agent’s.
Jujutsu setup: jj 0.42.0, a colocated repository (jj and git on the same working copy), and the most-installed external jj skill, all in place before timing.
Repeated runs per cell: Each agent–tool–scenario cell runs several times (k, shown above); the figures are means over those runs, with a Wilson 95% interval on each pass rate.

k=10n=60 per tool–agentgrader: git-statejj 0.42.0360 runs

About

About this benchmark

Provenance models · tools · hashes

Codex: gpt-5.5 · codex-cli 0.144.6
Claude: claude-opus-4-8 · 2.1.215 (Claude Code)
grader: deterministic git-state verifier · no LLM judge
GitButler: build 3f654efed9 · skill 8acc12b42c
Jujutsu: jj 0.42.0 · build 849c9ab4bb · onevcat/skills@onevcat-jj
batch: k=10 · 360 runs · 2026-07-20 · results.json ↗ · source ↗

Results history 10 batches

2026-07-20 · k=10 · 6 scenarios · Opus 4.8 · GPT-5.5 · 359/360 · shown above
2026-07-06 · k=10 · 5 scenarios · Opus 4.8 · GPT-5.5 · 299/300
2026-07-05 · k=8 · Opus 4.8 · GPT-5.5 · 239/240
2026-07-03 · k=7 · Opus 4.1 · GPT-5.5 · 193/210
2026-07-01 · k=5 · 3 tools · 2 agents · 137/150
2026-06-29 · k=5 · 3 tools · 2 agents · 142/150
2026-06-28 · k=5 · git + GitButler · 100/100
2026-06-27 · k=5 · git + GitButler · 99/100
2026-06-26 · k=5 · git + GitButler · 95/100
2026-06-22 · k=3 · git + GitButler · 60/60

Related benchmarks 4

Most benchmarks here hold the tool constant and compare models; this one holds the agents constant and compares the tools: the question a team faces once the agent is chosen.