indexdesk 1.0.0 Faceted full-text search for local file trees Techlosoft - Search Intelligence product line ------------------------------------------------------------------------------- WHAT IT IS ------------------------------------------------------------------------------- indexdesk builds a persistent inverted index over a directory tree and stores per-document facets - extension, size and modification time - right next to the term postings. That is the difference from a plain full-text indexer. Once the index exists you can narrow a text query by metadata (only .go files, only files over 100 KiB, only files touched in the last week) and you can ask for FACET COUNTS, which show you the shape of a result set before you drill into it: how many hits are .md, how many fall in each size band, how many are older than a year. Everything is Go standard library. One static binary, no runtime dependencies, no daemon, no network access. The index is a single JSON file you can move, copy, diff or delete. ------------------------------------------------------------------------------- INSTALL ------------------------------------------------------------------------------- Pick the binary for your platform from dist/ and put it on your PATH: dist/indexdesk-linux-amd64 dist/indexdesk-darwin-arm64 (Apple Silicon) dist/indexdesk-darwin-amd64 (Intel Mac) dist/indexdesk-windows-amd64.exe Or build from source (Go 1.24+, no network needed): go build -o indexdesk . ------------------------------------------------------------------------------- QUICK START ------------------------------------------------------------------------------- # index a tree once indexdesk build ~/projects --index ~/.cache/projects.json # query it as many times as you like - the index is never rewritten indexdesk query timeout --index ~/.cache/projects.json # narrow by facets indexdesk query timeout --index ~/.cache/projects.json --ext .go --newer-than 7d # see the shape of the result set indexdesk query timeout --index ~/.cache/projects.json --facets # see the shape of the whole corpus indexdesk facets --index ~/.cache/projects.json ------------------------------------------------------------------------------- COMMANDS ------------------------------------------------------------------------------- indexdesk build --index [--max-size 5MB] [--verbose] Walks recursively. Every text file is tokenized (split on every non-alphanumeric character, lowercased) into an inverted index, and its facets are recorded: extension, size in bytes, modification time. --max-size N Skip files larger than N. Default 5MB. Accepts plain byte counts and suffixes: 900, 64K, 5MB, 2GiB. KB and KiB are both 1024 bytes. --verbose, -v List every skipped file and the reason it was skipped. Skipped automatically: files over --max-size, files with a known binary extension (.png .zip .pdf .exe .docx and about 40 more), files whose first 8 KiB contain NUL bytes or otherwise do not look like text, and the .git directory. Nothing else is filtered - dotfiles and dot-directories other than .git are indexed. Writing is atomic: the index goes to .tmp and is then renamed. indexdesk query --index [filters] [--facets] [--limit N] [--json] Returns documents containing ALL of the terms (AND), then applies the facet filters to that set. --ext .go,.md Keep only these extensions. A leading dot is optional (.go and go both work). The literal value "none" selects files that have no extension. --min-size N Keep only documents of at least N bytes. --max-size N Keep only documents of at most N bytes. --newer-than 7d Keep only documents modified inside the window. Units: m (minutes), h (hours), d (days), w (weeks). A bare number means days. --facets Print counts per extension, per size band and per age band for the result set. Counts always cover the whole result set, even when --limit truncates the listing. --limit N Print at most N results. 0 (default) prints all. --json Machine readable output. Query terms are tokenized the same way as documents, so case does not matter and punctuation is ignored. A query that matches nothing prints a plain "no matches" line and exits 0. That is a normal empty result, not an error. indexdesk facets --index [--json] Prints the facet distribution of the entire index - extension, size band and age band counts across every indexed document - without running a query. indexdesk help | --help | -h Usage. Exits 0. indexdesk version Version. Exits 0. ------------------------------------------------------------------------------- CONVENTIONS ------------------------------------------------------------------------------- Flags may appear before or after positional arguments. These are identical: indexdesk query alpha beta --index i.json --facets indexdesk query --index i.json --facets alpha beta Exit codes: 0 success, including a query that matched nothing 1 bad invocation, missing or unreadable index, unreadable directory Usage text goes to stderr on a bad invocation and to stdout when help was asked for explicitly. Size buckets: <1 KiB, 1-10 KiB, 10-100 KiB, 100 KiB-1 MiB, >=1 MiB Age buckets: <1d, 1-7d, 7-30d, 30-365d, >=365d ------------------------------------------------------------------------------- WHAT IS IMPLEMENTED ------------------------------------------------------------------------------- - Recursive directory walk with binary and oversize file skipping. - Inverted index over lowercased alphanumeric tokens. - Per-document facets stored in the index: extension, size, mtime. - Persistent single-file JSON index. build writes it; query and facets open it read-only and never modify it, so you can build once and query as many times as you like from separate processes, scripts or shells. - Multi-term AND matching. - Facet filtering of results by extension, minimum size, maximum size and modification age. - Facet counts for a result set (--facets) and for the whole corpus (the facets command). - JSON output for query and facets. - Atomic index writes and clear, actionable errors for a missing, corrupt or wrong-version index. ------------------------------------------------------------------------------- WHAT IS NOT IMPLEMENTED ------------------------------------------------------------------------------- These are genuinely absent in 1.0.0. Do not expect them to work. - Incremental re-indexing. There is no change detection and no watch mode. If files change you must re-run build over the whole tree; it rewrites the index from scratch every time. - Content extraction from binary document formats. PDF, DOCX, XLSX, PPTX, ODT and similar files are skipped entirely, not parsed. Their text is not searchable. Only plain text files are indexed. - Phrase and proximity queries. "foo bar" in quotes is not a phrase search, and there is no NEAR operator. Multiple terms are always a plain AND over whole documents; term positions are not stored, so the tool cannot tell whether two terms were adjacent or a thousand lines apart. - Relevance ranking. There is no scoring of any kind - no TF-IDF, no BM25, no term frequency weighting. A document either matches or it does not, and results are listed in alphabetical path order. --limit therefore truncates alphabetically, not by "best" match. - Boolean OR and NOT, wildcards, prefix search, stemming, fuzzy matching and synonyms. A term matches only its exact lowercased alphanumeric form. - Snippets and match context. Results are file paths with facets; the tool does not show you the matching line or where in the file the term occurred. - Concurrency. Indexing is single threaded, and the index is loaded fully into memory for each query, which bounds practical corpus size to what fits in RAM. ------------------------------------------------------------------------------- NOTES ------------------------------------------------------------------------------- Index format is JSON, version 1, and is checked on load. An index written by a future version, or by another tool, is rejected with a message telling you to rebuild rather than being silently misread. Index size is roughly proportional to the number of distinct terms in the corpus, not to the corpus size in bytes.