How to search your files by what is inside them

Why the built-in search misses things

Windows Search and Spotlight both work from an index, and both decide what goes into it. Between them the usual exclusions are: files in folders marked as excluded, files above a size limit, file types with no registered content handler, anything on a drive that was not indexed, network locations, and — the one that catches most people — anything the indexer has not got to yet.

None of this is announced. A search returns some results, and there is no way to distinguish “not found” from “not indexed”.

The second problem is that the index gets confused. On both systems the standard advice for “search stopped working” is to rebuild the whole thing, which takes hours and is an odd thing to have to do to a search box.

Two honest approaches

Read the disk every time. Walk the tree, open the files, look for the pattern. Always correct, always current, never surprises you. On a project folder it is instant. On a whole home directory it is slow, because it genuinely is reading everything.

Build an index once, query it many times. Walk the tree, tokenise the text, write an inverted index — a map from each word to the files containing it. Queries are then near-instant and never touch the filesystem being searched. The trade is that results are as fresh as the last build, and you know exactly when that was.

The right choice is not about which is better. It is about the ratio of searches to changes. Searching a source tree you edit constantly: read the disk. Searching ten years of documents that hardly change: index.

What your machine already does

grep -ril "quarterly forecast" ~/Documents            # macOS / Linux

Select-String -Path C:\Users\me\Documents\*.* -Pattern "quarterly forecast" -List

ripgrep is dramatically faster than either if you are willing to install it, and respects .gitignore, which for a code tree is the difference between a search and a wait.

These do not read inside PDFs, Word documents or spreadsheets, because those are containers rather than text. That is the main thing a purpose-built tool adds.

Narrowing is worth more than matching

The single most useful feature in a file search is not better matching. It is being able to say: of those hits, show me only the .go files, under 100 KB, changed this year.

A search that returns four hundred results and no way to cut them down has not really answered the question. If a tool records the type, the size and the date alongside the words, you can whittle four hundred to six without re-running anything.

The programs for this

• FindPilot is the no-index one. search takes a root and matches on --name as a glob, --content as a pattern, or both, with --regex and --ignore-case. There is nothing to build first and nothing to go stale.
• SearchForge is the index-once one. index builds an on-disk inverted index and updates it incrementally on later runs; queries then run against the index without re-walking or re-reading the tree at all.
• IndexDesk adds the narrowing. It records the facets of every file as it indexes, so a query can be filtered by type, size or date, and facets shows you what the index knows before you ask.
• LocalLens searches several machines’ indexes at once, which answers “who else has this file” as well as “where is it”.

Free while we are in preview, one file each, Windows and Mac.

Where indexes go wrong

An index of a folder you are actively working in is out of date within minutes, and a stale hit is worse than no hit — you go to the file and the text is not there any more.

So: index things that are finished. Archives, references, delivered work, a documents folder. Read the disk for anything live. And when a tool builds an index, it should be able to tell you when it was built, because “the index says no” and “the file does not contain it” are different answers and you need to know which one you got.