How to find duplicate files on Windows and Mac

The three kinds of duplicate

Almost every complaint about a duplicate finder comes down to using one built for a different kind of duplicate than the one you have.

Identical bytes. Two files with the same contents, whatever they are called. A copy made with Ctrl+C, a folder restored twice from a backup, a photo library imported into two places. These can be found with total confidence: hash both files, compare the hashes, and there is no judgement call left.

Same file, different name. invoice.pdf, invoice (1).pdf, invoice_final.pdf. The bytes are often not identical — a PDF re-saved by a different reader, a document exported twice — so hashing finds nothing at all. What these share is the name, and the only way to catch them is to compare names.

Same picture, different file. A photo resized for email, re-exported at a different quality, or downloaded again from a service that recompressed it. Every byte is different and the names usually are too. Finding these means comparing what the images look like, not what they contain.

A tool that reports “no duplicates found” is often just answering a different question from the one you asked.

Try it without installing anything first

For a few thousand files, your computer already has enough.

On Windows, PowerShell will group files by size, which is where almost every duplicate pair shows up:

Get-ChildItem -Recurse -File | Group-Object Length |
  Where-Object Count -gt 1 | Sort-Object Count -Descending

On macOS, checksum everything and look for repeats:

find . -type f -exec shasum {} + | sort | uniq -w40 -d

Both are fine for a project folder. Both fall over on a whole drive: the macOS one reads every byte of every file, and neither tells you how much space you would actually get back, which is the number that decides whether any of this is worth doing.

Why size comes first

This is the difference between a scan that finishes and one you abandon.

Two files of different sizes cannot possibly have the same contents. So there is no reason to read a file at all until you know something else on the disk is exactly the same length. Group everything by size, throw away every group with one member, and only then start hashing. On a typical drive that skips well over ninety per cent of the data — most of your disk never gets read.

If a tool takes hours on a drive you know is mostly unique files, it is hashing first and grouping second.

Nothing should be deleted

The finding is the easy part. The part that goes wrong is the acting.

Whatever you use, it should do two things. It should show you the whole plan before it touches anything — every file, where it is going, how much space it frees. And when you say yes, it should move the files somewhere, not unlink them. A folder of quarantined duplicates that you delete yourself in a month is recoverable at every point. A file that has been unlinked is a data-recovery job.

Be especially careful with anything offering to delete “all but one” across a whole drive automatically. Program installations, source trees and game data are full of legitimately identical files — licence texts, empty __init__.py files, default configs — and moving one of a pair breaks the software that expected it.

Which one you need

• DupePilot compares contents. It groups by size, hashes only the files that share a size with something, and reports sets sorted by how much space you would reclaim. quarantine moves everything but the oldest copy into a folder you name, and prints the plan and stops unless you add --apply.
• DuplicateDeck compares names. It normalises filenames and clusters them by edit distance, so the invoice (1).pdf family turns up even though the bytes differ. It never opens a file.
• PhotoSweep compares pictures. It computes a perceptual hash of every image and groups the ones that look the same, which is the only one of the three that catches a resized re-export.
• CleanGallery answers a different question: across several drives or shares, which one is wasting the most room? It is read-only — it never moves anything — and ranks targets worst-first so you know where to start.

All four are free while we are in preview, and each is a single file you run: no installer, and no account.

What to keep

When you have a set of identical files, the useful default is to keep the oldest one. It is most likely the original, most likely the one other things point at, and most likely to be in the folder you meant it to be in. Copies made later tend to live in Downloads, on the desktop, or in a folder called new.

The exception is when the oldest copy sits on a drive you are about to retire. Then keep the one you will still have next year.