# SearchForge (CLI prototype)

SearchForge is a working prototype of a persistent, on-disk inverted-index
search engine: you build an index once (a potentially slow, infrequent
operation that walks a directory tree and tokenizes every file), and then
run as many queries as you like against that stored index near-instantly,
since a query is nothing more than a map lookup in the index file - no
re-walking the directory, no re-reading the files, per query. That is the
real "why pay more for Pro" differentiator over the sibling tool FindPilot,
which does a live, unindexed scan of the whole tree on every single search.
The index format and the incremental-update logic below are the genuinely
different, more sophisticated mechanism this tier is built around.

Index format: the index is a single JSON file with two top-level maps. The
"tokens" map goes from a lowercase word to the list of file paths that
contain it (the inverted index proper). The "files" map goes from each
indexed file's path to its metadata: size, modification time, when it was
last (re)indexed, and the exact list of tokens that file itself
contributed. Storing each file's own token list is what makes incremental
updates and removals exact and cheap - to update or remove a file, the
indexer looks up that file's previously stored tokens and surgically
removes just its path from just those token buckets, instead of rebuilding
the whole index from nothing.

Tokenization rule: a file's filename is always tokenized; a text file's
content is tokenized too. Binary files (detected by the presence of a NUL
byte anywhere in the first 8KB, the same heuristic FindPilot uses) are
indexed by FILENAME ONLY - their content is never read for tokenization.
Tokenizing means: lowercase the text, then split it into maximal runs of
Unicode letters and digits, treating every other character (whitespace,
punctuation, symbols) as a separator; empty tokens are dropped and a file's
duplicate tokens are collapsed to one entry.

Incremental updates: running "index" against an existing index file without
--rebuild compares each file's current (size, mtime) to what is stored in
the index. Unchanged files are left completely alone (not re-tokenized, not
re-read). New or changed files are (re)tokenized and their entries updated.
Files that no longer exist on disk are removed from the index, including
removing their contributed tokens from the token map. --rebuild ignores any
existing index content and rebuilds everything from scratch.

OCR indexing (for scanned/image content), a quick-launcher UI, and saved
queries from the full product concept are on the roadmap - this prototype
implements the core index-and-query engine only. See ../plan.md for the
full product plan.

## Build from source

Requires Go 1.24+, no external dependencies.

    go build -o searchforge .

Cross-compile for another platform:

    GOOS=windows GOARCH=amd64 go build -o searchforge.exe .
    GOOS=darwin  GOARCH=arm64 go build -o searchforge .

## Usage

  searchforge index <dir> --index <indexfile> [--rebuild]
      Walk <dir> and build/update the inverted index at <indexfile>.
      Without --rebuild, an existing index is updated incrementally:
      unchanged files (same size+mtime) are left alone, new/changed files
      are (re)tokenized, and files that no longer exist on disk are
      removed from the index. --rebuild ignores any existing index
      content and rebuilds from scratch. Prints a summary: files added,
      updated, removed, unchanged; total files; total unique tokens; time
      taken; and the resulting index file size.

  searchforge query <term...> --index <indexfile> [--json] [--max N]
      Look up one or more space-separated terms in the index. Multiple
      terms are treated as AND: a file must contain every given term to
      match (implemented as a set-intersection across each term's file
      list). This is a direct map lookup against the stored index file -
      it does not touch, walk, or re-read the filesystem that was
      indexed. --max limits how many results are printed (default 50).
      --json prints a structured JSON object ({terms, count, matches,
      truncated}) instead of plain text. Prints matching paths (up to
      --max) plus a total match count.

  searchforge stats --index <indexfile> [--json]
      Reports index metadata: indexed file count, total unique tokens,
      index file size on disk, and when the index was first built and
      last updated (RFC3339 timestamps). --json prints the same data as
      a structured JSON object.

  searchforge help
      Show usage. Also available as -h / --help, or as the first
      argument to any subcommand.

Flags accept either "-flag value" or "--flag value" form.

Index file format (JSON), for reference:

    {
      "root": "<directory that was indexed>",
      "built_at": <unix seconds of first build>,
      "updated_at": <unix seconds of most recent build/update>,
      "tokens": { "<token>": ["<path>", ...], ... },
      "files": {
        "<path>": {
          "size": <bytes>,
          "mtime": <unix seconds>,
          "indexed_at": <unix seconds>,
          "tokens": ["<token>", ...]
        },
        ...
      }
    }

## Prebuilt binaries

See ../downloads/ for prebuilt binaries (Windows/macOS/Linux) and
CHECKSUMS.txt for their SHA-256 hashes. These are unsigned indie builds -
Windows SmartScreen and macOS Gatekeeper will warn on first run, which is
expected until a code-signing certificate is in place.