How to download a large file without it arriving corrupt
A download that stops at 94 per cent and one that finishes and will not open are the same underlying problem, and one of them is much harder to notice.
Short answer
Check the hash the publisher printed. Resuming a download is safe as long as the server confirms the file has not changed since you started — and if it will not confirm that, starting again is cheaper than debugging a corrupt file later.
How resuming actually works
An HTTP resume is one header. The client says Range: bytes=1048576- — send me everything from a megabyte in — and the server answers 206 Partial Content and does exactly that.
Two things have to be true for that to be safe.
The server must support it. Many do, some do not, and a server that ignores a Range header replies 200 OK with the whole file. A client that appends that to what it already had produces a file the right length in the wrong order, which is exactly the kind of corruption nobody notices until later.
And the file must not have changed. This is what ETag and Last-Modified are for: the client remembers them from the first attempt and sends If-Range on the second, and the server either continues or starts over. Without that check, resuming a file that was replaced on the server halfway through your download splices two different releases together.
Any resuming client worth using does the If-Range check. It is the difference between resume and “hope”.
Why several connections is faster, and when it is not
Downloading a file in segments — several ranges at once, reassembled — is faster in specific cases and pointless in others.
It helps when a single connection is the limit rather than your line: a per-connection rate cap at the far end, a long-distance link where TCP takes a while to get up to speed, or a server that is fine in aggregate and slow per stream.
It does nothing when your own connection is already saturated. Four segments each at a quarter of the speed is the same download, four times the complexity.
And past about eight segments it starts costing you. Servers begin refusing, or rate limiting the address, and reassembly overhead grows. Four is a reasonable default; sixteen is a good way to be blocked.
Verify the bytes, not the size
The size being right proves almost nothing. Truncation shows up in the size; a flipped bit, a proxy that mangled the stream, a resume that appended the wrong range, and a mirror that is quietly serving an old build do not.
Publishers print a SHA-256 for exactly this. Check it:
shasum -a 256 file.iso # macOS / Linux
Get-FileHash file.iso # Windows PowerShell
Compare the entire string. For anything you are going to execute, this matters twice over: it is also the check that the mirror gave you what the publisher published.
Downloading a whole list politely
Fetching two hundred URLs from the same host as fast as your connection allows is how you get rate limited, and then blocked, and then have to explain yourself.
What keeps you welcome is a limit per host rather than in aggregate: a cap on requests per second and a cap on simultaneous connections, applied per hostname, so a list spanning several sites goes quickly overall while no single site sees a burst. Respect Retry-After when a server sends it, and back off rather than retrying immediately.
This is politeness and it is also self-interest — a job that finishes slowly beats one that gets you banned at item 40.
The programs for this
- GrabFlow is the single-file one: segmented download, resume, and a SHA-256 check against a hash you pass in, so verification is part of the download rather than something you remember afterwards.
- DownloadPilot keeps a queue on disk. Items are added to a queue file, workers drain it concurrently, and resuming after a reboot or a crash is just running it again — the queue is the state.
- DropDeck takes a manifest of URLs and works through it with a real per-host token bucket and a per-host concurrency cap.
planshows what it would do before it does it. - FetchForge is the team version: one server holds one queue, teammates submit to it from their own machines, and a pool of workers drains it into a shared output directory.
Free while we are in preview, one file each, Windows and Mac.
If it keeps failing
Try a different mirror before you try a different tool. Repeated failures at roughly the same point are usually the far end, not you.
If it fails at exactly the same byte every time, that is a server or a proxy problem, and no amount of retrying will move it.
And check the disk has room. A download that dies at 94 per cent on a nearly full drive is not a network problem, and the error message rarely says so.