How to copy a large folder and know every file arrived

What “copy finished” actually means

When a file copy completes, what you know is that the operating system accepted every byte you handed it. What you do not know is that those bytes are on the destination disk, correct and readable.

Between the two sit several places things go wrong: a write cache that had not flushed when the drive was unplugged, a USB enclosure that silently drops bytes under load, a network share that timed out on one file out of eighty thousand, a destination drive with bad sectors it has not remapped yet, a filesystem that quietly refused a filename it did not like.

None of these produce an error message you will see. They produce a folder that looks right, has roughly the right size, and contains a handful of files that will not open — which you find out in a year.

The check that matters

Copy the files. Then read every file back off the destination disk and compare its checksum with the source.

The “off the disk” part is the one people get wrong. Comparing a file against a copy that is still in the operating system’s cache compares memory against memory and proves nothing about what landed. A verification pass has to actually re-read from the destination device.

By hand, for a folder you care about:

cd /source && find . -type f -exec shasum -a 256 {} + | sort > /tmp/src.txt
cd /dest   && find . -type f -exec shasum -a 256 {} + | sort > /tmp/dst.txt
diff /tmp/src.txt /tmp/dst.txt

Empty output means every file arrived intact. It is slow — it reads everything twice — and for a one-off transfer of something irreplaceable it is time well spent.

Moving is where the risk is

Copying is safe: if it goes wrong, you still have the original. Moving is not, and the ordering is everything.

A safe move is: copy the file, re-read the copy, verify it byte for byte, and only then remove the source. If verification fails, leave that file’s source completely alone and report it. That way an interrupted move leaves you with some files in both places — inconvenient, entirely recoverable — rather than some files in neither.

An unsafe move deletes the source when the write call returns. On the same filesystem this is fine, because it is only a rename and no data is going anywhere. Across two devices it is a gamble taken on your behalf, once per file.

Long paths and odd characters

Two things break large transfers routinely and are worth knowing about before rather than during.

Windows path length. The classic limit is 260 characters, and copying a deeply nested tree into a destination one level deeper than the source is enough to cross it. The copy fails partway with a message about the filename being too long.

Characters one side allows and the other does not. : and ? are legal on macOS and Linux and not on Windows. A file called Notes: draft?.txt copies from a Mac to an exFAT drive by being silently renamed, or by failing.

Both are much easier to handle when a dry run lists the problems before the transfer starts than when the transfer stops at file 40,000.

The programs for this

• CopySure copies a tree, then re-reads every file off the destination disk and checks it byte for byte. diff compares two trees that already exist.
• MoveGuard does the safe move: a source file is only ever deleted after its destination copy has been re-read and hash-verified identical, and if verification fails for a file, that file’s source is left completely untouched. --dry-run lists every file that would move and reports the problems the real run would hit.
• SyncProof pushes one source tree to several destinations in a single pass and then verifies each, so you find out which destination came out bad rather than that something did.
• TransferForge is the same verified transfer with a client-deliverable manifest: what was copied, how big it was, and the checksum of each file, grouped by media type. It resumes if it is interrupted.

Free while we are in preview, one file each, Windows and Mac.

When the source is about to disappear

Offloading a camera card, clearing a drive you are returning, decommissioning a machine — this is the case where verification stops being optional, because there is about to be exactly one copy of everything.

Verify before you format. Not after.