DuplicateDeck 1.0.0 Near-duplicate file finder that matches on FILENAME, not on file contents. Part of the Techlosoft "Duplicate + Photo Cleanup" line. WHY THIS TOOL EXISTS ==================== Content-hash dedupers (like our own DupePilot) compare bytes. They are exact and fast, and they find nothing at all in this very common situation: invoice.pdf invoice (1).pdf invoice_final.pdf invoice - Copy.pdf Those four files are the same document to a human, but every one of them has a different SHA-256, because a re-download, a re-save, a re-export or an "edit one word" pass changes the bytes. A hash deduper reports zero duplicates. A perceptual image hasher (PhotoSweep) cannot help either, because these are not images. DuplicateDeck attacks the problem from the other side: it normalizes the FILENAME and measures how far two names are apart using a real Levenshtein edit distance. Name-families like the one above collapse into a single cluster, with a suggested keeper, and you can quarantine the rest. This is the opposite trade-off from a content deduper. It is fast (it never reads file contents at all) and it catches families that hashing misses -- but a name match is a HINT, not proof. Review a cluster before you apply it. WHAT IS IMPLEMENTED =================== * Recursive directory walk (regular files only; symlinks and devices are skipped, unreadable paths are reported as warnings and do not abort). * Filename normalization: - take the base name, drop the extension - lowercase - strip common copy-markers from the end (and "Copy of " from the front): "(1)" "[2]" " - Copy" "-copy" "_copy" "copy 2" "_final" "-draft" " new" "_old" "orig" "original" "edited" "dup" "duplicate" "backup" "bak" "v2" - collapse every remaining run of punctuation/whitespace to one space, keeping Unicode letters and digits intact ("Müller-Bericht" becomes "müller bericht", "résumé" and "写真" survive unharmed) The result is the "stem"; the markers that were removed are kept as a separate list. * Levenshtein edit distance, implemented here from scratch: classic dynamic programming with the two-row space optimization, operating on runes so non-ASCII names behave correctly. Verified against an independent Python reference implementation, including the textbook cases kitten -> sitting = 3 and flaw -> lawn = 2. * Similarity score: distance = levenshtein(stem_a, stem_b) + |markers_a symmetric-difference markers_b| length = max(len(stem)+len(markers)) over the two names similarity = 1 - distance / length (clamped to [0,1]) Each stripped copy-marker costs exactly one edit. That is deliberate: it means "invoice" and "invoice (1)" score 0.875 rather than a flat 1.0, so --min-similarity stays a real dial. Loosen it to catch more families, tighten it to demand near-identical names. * Clustering: every pair above the threshold is joined with a union-find, so clusters are transitive. A file that is similar to two others pulls them all into one cluster even if those two are not similar to each other. Pairs whose lengths differ too much to possibly reach the threshold are skipped without running the DP. * Keeper selection: the largest file wins; a newer mtime breaks a size tie; path order breaks a full tie. Everything else in the cluster is a candidate for quarantine. * Quarantine, not deletion. Non-keepers are MOVED into a quarantine directory with their path relative to the scanned root preserved, so "archive/2023/invoice-copy.pdf" lands at "/archive/2023/invoice-copy.pdf". Undoing a mistake is a matter of moving files back. * Dry run by default. `quarantine` prints exactly what it would move and touches nothing until you add --apply. * Never overwrites: if something already sits at the destination path, the incoming file is stored as "name~1.ext", "name~2.ext" and so on. * A quarantine directory located inside the scanned tree is skipped during the walk, so re-running the command does not re-cluster files you already quarantined. * JSON output (--json) for scan, quarantine and compare. KNOWN FALSE POSITIVES -- READ THIS BEFORE USING --apply ======================================================= A name match is a hint, not proof. The tool is DESIGNED to be a bit greedy, because a dry run and a quarantine folder make a wrong guess cheap. Two patterns will bite you if you apply blindly: * SEQUENTIAL CAMERA / SCANNER FILENAMES. "IMG_4471.JPG" and "IMG_4472.JPG" differ by one character and score 0.875 at default settings, so they cluster -- even though they are two completely different photos. Do NOT run DuplicateDeck across a camera roll at the default threshold. Either raise --min-similarity well above what one digit costs, or use PhotoSweep, which compares the actual images. The same applies to "page 01.png" / "page 02.png", "track 3.mp3" / "track 4.mp3" and dated files like "invoice 2024-01.pdf" / "invoice 2024-02.pdf". * SHORT NAMES. On a short stem a single edit is a large fraction of the length, but so is a single marker. "a.txt" and "b.txt" score 0; "ab.txt" and "ab (1).txt" score 0.667 and do NOT cluster at the default. Very short names behave erratically by nature -- check them with `compare`. Use `duplicatedeck compare ` to see exactly what a pair scores before you commit to a threshold, and always read the dry run. WHAT IS NOT IMPLEMENTED (ROADMAP) ================================= DuplicateDeck 1.0 never opens a file. It looks at names, sizes and mtimes only. In particular the following are NOT in this build: * Content-hash cross-check. A planned second pass would hash the members of a name-cluster and tell you whether a cluster is ALSO a byte-for-byte duplicate set, so you could auto-approve the certain cases and hand-review only the rest. Today the tool cannot tell you that. * Media metadata / EXIF grouping. Grouping photos by capture timestamp, camera body or GPS location -- so that "IMG_4471.JPG" and "beach sunset.jpg" are recognized as the same shot -- is planned and not present. Names that share no characters will never cluster here. * GUI review step. A visual side-by-side review pass before quarantining is planned. Today the only review is reading the scan output. * Fuzzy matching beyond edit distance: no token/word-order matching ("report final Q1" vs "Q1 report"), no phonetic matching, no learned similarity. Word order matters to Levenshtein. * No undo command. Undo is a manual move back out of the quarantine directory. (The tool never deletes, so undo is always possible.) * No parallelism. Comparison is O(n^2) pairs with a length pre-filter; it is comfortable on tens of thousands of files, not millions. * No archive/cloud/remote sources, no scheduling, no config file. USAGE ===== duplicatedeck scan [--min-similarity 0.8] [--json] duplicatedeck quarantine --quarantine [--min-similarity 0.8] [--apply] duplicatedeck compare [--json] duplicatedeck help | -h | --help duplicatedeck version Flags may be written before or after the positional arguments. --min-similarity F Cluster threshold in [0,1]. Default 0.80. 0.90+ demands near-identical names. --quarantine DIR Destination for the quarantine command. Required. --apply Actually move files. Without it, quarantine is a dry run and nothing is touched. --json Machine-readable output. Exit codes: 0 success, including "no clusters found" and an empty directory 1 bad invocation, unreadable root, or a failed move EXAMPLES ======== Look at a Downloads folder: duplicatedeck scan ~/Downloads Cluster 1 stem "invoice" 3 files reclaimable 130 B dup 81 B 2026-08-10 03:42 sim 0.75 invoice (1).pdf dup 49 B 2026-08-10 03:42 sim 0.88 invoice.pdf KEEPER 1.2 KiB 2026-08-10 03:42 sim 1.00 invoice_final.pdf Understand or tune the threshold before trusting it: duplicatedeck compare "invoice.pdf" "invoice (1).pdf" levenshtein(stem_a, stem_b) = 0 marker difference = 1 effective distance = 1 normalized length (max) = 8 similarity = 0.8750 Preview a cleanup (nothing is moved): duplicatedeck quarantine ~/Downloads --quarantine ~/dupe-quarantine Then perform it: duplicatedeck quarantine ~/Downloads --quarantine ~/dupe-quarantine --apply Check the quarantine folder, and when you are satisfied, delete it yourself. DuplicateDeck will not do that for you. BUILD FROM SOURCE ================= Go 1.24 or newer. Go standard library only -- no third-party dependencies, no network access needed to build. go build -o duplicatedeck . Cross-compile: GOOS=windows GOARCH=amd64 go build -ldflags="-s -w" -o dist/duplicatedeck-windows-amd64.exe . GOOS=darwin GOARCH=arm64 go build -ldflags="-s -w" -o dist/duplicatedeck-darwin-arm64 . GOOS=darwin GOARCH=amd64 go build -ldflags="-s -w" -o dist/duplicatedeck-darwin-amd64 . GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o dist/duplicatedeck-linux-amd64 . Prebuilt binaries for those four targets are in dist/. They are unsigned indie builds: Windows SmartScreen and macOS Gatekeeper will warn on first run until a code-signing certificate is in place. HOW IT RELATES TO THE OTHER TOOLS IN THE LINE ============================================= DupePilot exact SHA-256 content duplicates. Certain, but blind to renamed/re-saved variants. PhotoSweep perceptual aHash near-duplicate IMAGES. Finds visually similar photos regardless of name. DuplicateDeck name-similarity clustering via Levenshtein. Finds the "invoice (1)" families that the other two cannot see, for any file type, without reading a single byte. Running DupePilot first and DuplicateDeck second is a reasonable workflow: remove the certain duplicates, then review the probable ones.