How to find duplicate photos, including resized copies

Why the obvious tool finds nothing

Run a checksum-based duplicate finder over ten years of photos and it will very often come back with almost nothing. That is the correct answer to the question it was asked.

A photo that has been through anything at all — emailed, uploaded and downloaded again, exported at a smaller size, opened and saved by a phone app, backed up by a service that recompresses — is a different sequence of bytes from the original. Sometimes wildly different: re-encoding a JPEG at a different quality changes essentially every byte in the file while changing what you see hardly at all.

Checksums are built to notice a single flipped bit. That is exactly the wrong sensitivity here.

What to compare instead

The technique is called perceptual hashing, and the idea is simple enough to describe in a sentence: shrink the image down to a tiny grid, throw away the colour, and record for each cell whether it is brighter or darker than the average. You end up with a 64-bit fingerprint that depends on the shape of the picture and not on its size, its compression or its file format.

Two copies of the same shot then have fingerprints that differ in only a handful of bits. Count how many bits differ — the Hamming distance — and you have a dial. Zero means visually identical. Around five catches resizes and re-compressions. Turn it up much past ten and you start pulling in genuinely different photos that happen to share a composition, which on a library of holiday pictures means a lot of similar-looking beaches.

What this will and will not catch

It reliably catches: the same shot saved at two sizes, the same shot at two JPEG qualities, a PNG and a JPEG of the same image, a photo that has been through a messaging app.

It does not catch: two frames from the same burst. Those are genuinely different photographs of nearly the same scene, and no tool can tell you which one you meant to keep. If you shoot in burst mode, expect to make those calls yourself.

It also does not, on its own, tell you which copy is better. That is the next section.

Which copy to keep

When a group comes back, the useful order is:

1. Largest pixel dimensions. A 4000×3000 original beats a 1024×768 export every time, and file size is a bad proxy — a heavily compressed large image can be smaller on disk than a lightly compressed small one.
2. The one with intact metadata. Exports and re-uploads routinely strip EXIF, which means losing the date, the camera and often the location. A copy that still knows when it was taken is worth more than one that does not.
3. The one in the folder where it belongs. If one copy is in your library and one is in Downloads, keep the library.

Do this before you start

Photo libraries are the single worst place to run an automatic delete. Not because the matching is unreliable, but because the cost of one wrong call is a photograph nobody has another copy of.

So: whatever you use should move files to a folder rather than remove them, and it should show you the whole plan first. Keep the quarantine folder until you have browsed the library once and are satisfied nothing you wanted is missing. Then delete it in one go, when you can still put it back if you were wrong.

If your photos live in Apple Photos or Windows Photos, work on an exported copy or on the folders those apps read from, not on the library package itself. Rearranging a library’s internals underneath the app that manages it is a good way to end up with a database pointing at files that are no longer there.

The programs for this

• PhotoSweep is the one built for this job. It computes a 64-bit average hash for every .jpg, .jpeg, .png and .gif it finds and groups images within a Hamming distance you set — default five. --min-size keeps thumbnails and icons out of the results. clean moves everything but the first member of each group to a quarantine folder, and prints the plan and stops unless you add --apply.
• DupePilot is still worth one pass first. Real byte-identical copies do exist in most libraries — an import that ran twice — and those are the cases you can clear with total confidence before you start making judgement calls.
• DuplicateDeck catches the IMG_4021.jpg / IMG_4021 (1).jpg family, where the names are the giveaway and the bytes may not match.
• CleanGallery tells you which of several drives or shares is wasting the most on duplicates, if the library is spread across more than one place.

Free while we are in preview, one file each, Windows and Mac. More on the general case in how to find duplicate files.