Cloning large repos over HTTP is very slow

[royneary]

As mentioned in jonashaag/klaus#230 I tried to clone a large repo over HTTP using klaus and ran into a timeout.

Now I tried to clone using dulwich web-daemon directly and cloning the nixpkgs repo from localhost took 36 minutes. Here's the console output:

$ dulwich web-daemon
2018-11-25 11:59:30,155 INFO: Sending capabilities: [b'multi_ack_detailed', b'multi_ack', b'side-band-64k', b'thin-pack', b'ofs-delta', b'no-progress', b'include-tag', b'shallow', b'no-done', b'symref=refs/remotes/origin/HEAD:refs/remotes/origin/master', b'symref=HEAD:refs/heads/master']
2018-11-25 11:59:30,190 INFO: "GET /home/christian/projects/nixpkgs/info/refs?service=git-upload-pack HTTP/1.1" 200 20916
2018-11-25 11:59:30,195 INFO: Handling service request for git-upload-pack
2018-11-25 11:59:30,210 INFO: Client capabilities: [b'multi_ack_detailed', b'no-done', b'side-band-64k', b'thin-pack', b'ofs-delta']
2018-11-25 12:35:28,218 INFO: "POST /home/christian/projects/nixpkgs/git-upload-pack HTTP/1.1" 200 7810989836

The git client displayed the remote: dul-daemon says what line after around 10 minutes. The client side output:

git clone http://localhost:8000/home/christian/projects/nixpkgs
Cloning into 'nixpkgs'...
remote: dul-daemon says what
remote: counting objects: 1315755, done.
Receiving objects: 100% (1315755/1315755), 7.26 GiB | 4.44 MiB/s, done.
remote: how was that, then?
Checking out files: 100% (17682/17682), done.

This was done on a machine with a Core i5-6300HQ, I didn't bother to wait for it on my server (an old Atom N270). That's definitely not viable for web applications.

I'm using dulwich 0.19.6.

[jelmer]

On Sun, Nov 25, 2018 at 03:37:15AM -0800, Christian Ulrich wrote: As mentioned in jonashaag/klaus#230 I tried to clone a large repo over HTTP using klaus and ran into a timeout. Now I tried to clone using `dulwich web-daemon` directly and cloning the [nixpkgs](https://github.com/nixos/nixpkgs) repo from localhost took 36 minutes. Here's the console output: ``` $ dulwich web-daemon 2018-11-25 11:59:30,155 INFO: Sending capabilities: [b'multi_ack_detailed', b'multi_ack', b'side-band-64k', b'thin-pack', b'ofs-delta', b'no-progress', b'include-tag', b'shallow', b'no-done', b'symref=refs/remotes/origin/HEAD:refs/remotes/origin/master', b'symref=HEAD:refs/heads/master'] 2018-11-25 11:59:30,190 INFO: "GET /home/christian/projects/nixpkgs/info/refs?service=git-upload-pack HTTP/1.1" 200 20916 2018-11-25 11:59:30,195 INFO: Handling service request for git-upload-pack 2018-11-25 11:59:30,210 INFO: Client capabilities: [b'multi_ack_detailed', b'no-done', b'side-band-64k', b'thin-pack', b'ofs-delta'] 2018-11-25 12:35:28,218 INFO: "POST /home/christian/projects/nixpkgs/git-upload-pack HTTP/1.1" 200 7810989836 ``` The git client displayed the `remote: dul-daemon says what` line after around 10 minutes. The client side output:

Are you sure you're using dulwich 0.19.6? That message is gone in 0.19.6. Cheers, Jelmer

…

-- Jelmer Vernooij <[email protected]> PGP Key: https://www.jelmer.uk/D729A457.asc

[royneary]

Are you sure you're using dulwich 0.19.6? That message is gone in 0.19.6.

That's what my package manager (nix) says. The package definition says, it's using 0.19.6 from Pypi.

[jelmer]

This should be fixed in 0.21.0 (or at least significantly better). We now reuse existing deltas.