routewatch 1.0.0 - multi-site latency matrix ============================================ A probe agent runs at every office. Each agent measures the latency from ITS site to a list of targets and writes a report file. routewatch merges every site's report into one grid: rows = the reporting sites columns = the targets cells = the latency of that one site->target link The grid is the point. One machine can only tell you that IT is slow. Several vantage points measuring the same targets tell you WHOSE fault it is: a whole slow ROW -> that site's own uplink is bad a whole slow COLUMN -> the target itself is bad, everybody sees it one hot CELL -> only the path between those two endpoints is bad routewatch names which of those three it is, in words, every time it prints a matrix. That distinction is the entire reason this tool exists and it cannot be answered from a single machine's measurements no matter how good they are. WHAT IT MEASURES - AND WHAT IT DOES NOT --------------------------------------- IT MEASURES TCP CONNECT LATENCY. Each sample opens a TCP connection to host:port, times how long the connect took to complete, and closes it. That is all a cell in the grid ever contains. IT IS NOT ICMP PING. There is no ping here and there never will be without a redesign. Sending real ICMP echo requests needs a raw socket, which needs root or CAP_NET_RAW. routewatch is an ordinary unprivileged program, so it uses the one latency measurement any user can take: the TCP handshake. The numbers are therefore not comparable to ping output. A TCP connect includes the handshake round trip plus whatever the target's accept queue and kernel add, so it will normally read higher than ICMP to the same host, and it can be slow for reasons that have nothing to do with the network (an overloaded server that is slow to accept looks exactly like a slow link from here). IT DOES NOT SEE THE NETWORK PATH. routewatch cannot tell you WHICH HOP is slow. It has no traceroute, no TTL games, no per-hop timing. It measures the end-to-end link and nothing between the ends. When it says "site B is slow to everything", it means every measurement taken from site B was slow - it does not and cannot mean "site B's second-hop router is at fault". Finding the hop is a separate job that this version does not do. REPORTS ARE FILES, NOT A NETWORK PROTOCOL. There is no server, no agent daemon, no push, no pull, no authentication and no wire format. Each agent writes a JSON file; you collect those files however you already move files (scp, rsync, a shared mount, an object store, a git repo, email) and point routewatch at them. Collection is deliberately out of scope, which also means routewatch cannot tell you that a site stopped reporting - a site that never sends a file is simply absent from the grid, not marked as failed. A REPORT IS A SNAPSHOT. probe runs once and exits. There is no scheduler and no history. Two reports written hours apart merge into the same grid with no warning that they describe different moments; keep your collection interval tight and treat the grid as "roughly now". COMMANDS -------- routewatch probe --site --targets --out [--samples 5] [--timeout 3s] [--json] The agent command, run at each site. Takes --samples TCP-connect latency measurements per target and records min, median, p95, max and loss for each site->target pair, then writes the report. routewatch matrix [report2.json ...] [--dir ] [--metric median|p95] [--json] Merge every site's report into the grid, with per-row and per-column summaries and the row-vs-column verdict. routewatch worst [--dir ] [--top N] [--metric median|p95] [--json] The N worst site->target links, ranked, with their numbers. routewatch help | -h | --help Show help (exit 0) routewatch version | -v | --version Show the version TARGET LIST ----------- { "targets": [ {"name": "dc-east", "addr": "10.0.0.10:443"}, {"name": "dc-west", "addr": "10.1.0.10:443"} ] } A bare JSON array of the same objects also works, and "target" is accepted as an alias for "addr". The NAME is what lines the grid up across sites; the ADDRESS may legitimately differ per site, because each office may reach the same logical service through its own address. Give the same service the same name at every site or it will appear as two separate columns. STATISTICS ---------- Percentile definition, used everywhere in the program including the median: NEAREST RANK over the ascending-sorted list of SUCCESSFUL samples rank = ceil(p / 100 * n) clamped to [1, n] value = sorted[rank - 1] median = percentile(50) p95 = percentile(95) There is no interpolation, so every figure reported is one of the raw samples and can be found verbatim in raw_samples_ms in the report. With the default --samples 5 the p95 is simply the maximum; five samples do not contain a meaningful 95th percentile, so raise --samples if you want p95 to mean something. Failed samples are excluded from min/median/p95/max and counted separately as loss_percent. An unreachable or half-dead target therefore cannot distort the statistics of the reachable ones. A target with zero successful samples is recorded as reachable=false and is shown as a marker, never as a latency. READING THE GRID ---------------- 12.34 the chosen metric for that link, in milliseconds 12.34! same, but some samples were lost (see the report for loss_percent) X unreachable: every sample failed, 100% loss . that site did not probe that target at all X and . mean different things and are kept apart on purpose. X is a measurement that was taken and failed. A dot is a measurement that was never taken - the grid is the union of every report's targets, so a site that probes a shorter list leaves gaps rather than zeros. HOW THE ROW-VS-COLUMN VERDICT IS COMPUTED ----------------------------------------- The grid of reachable cells is decomposed by Tukey median polish: cell ~= overall + site effect + target effect + residual fitted by alternately subtracting row medians and column medians until it settles. Missing and unreachable cells take no part in the fit. Whichever term is largest names the fault: SITE-WIDE the largest term is one site's effect -> that site is slow to everything, suspect its uplink TARGET-WIDE the largest term is one target's effect -> everyone is slow to it, suspect the target SINGLE LINK the largest term is one cell's residual -> only that pair is bad, suspect the path between them UNIFORM nothing clears the noise floor, which is set at four times the upper quartile of the absolute residuals (or a quarter of the baseline, whichever is larger) Unreachable links outrank any amount of slowness and are reported first, with their own row/column reading: every site failing to reach one target is TARGET DOWN, one site failing to reach every target is SITE ISOLATED. The decomposition needs several sites and several targets to say anything. A one-row grid can only ever produce a target effect, because there is no second row to compare against - that is not a bug, it is the limit of the data. FAULT ISOLATION --------------- A report that cannot be read, does not parse, carries the wrong schema, has no site name or contains contradictory counts is SKIPPED with a specific reason printed to stderr (and listed under "skipped" in --json output). Every other report still merges. One corrupt file from one office never costs you the rest of the picture. If every input is skipped, routewatch exits 1 rather than printing an empty grid. If the same site name appears in two reports, the newer generated_at wins and a note says so. EXIT CODES AND OUTPUT --------------------- 0 success, including help and version 1 bad invocation (usage is printed to stderr) or a fatal error Help goes to stdout. Errors and the usage text that follows a bad invocation go to stderr. --json output is a single JSON document on stdout and is the supported machine interface for all three commands. EXAMPLES -------- # at each office routewatch probe --site hq --targets targets.json --out hq.json routewatch probe --site branch-a --targets targets.json --out branch-a.json # after collecting the files somewhere routewatch matrix --dir reports routewatch matrix --dir reports --metric p95 routewatch worst --dir reports --top 10 --json ROADMAP ------- * Traceroute-style path analysis, so a slow link can be attributed to a specific hop instead of only to its two endpoints. * Continuous scheduled probing with retained history, so the matrix can show trend and regression rather than a single snapshot. * A real agent transport, so reports reach the collector on their own instead of being copied around by hand, including detection of sites that have stopped reporting. BUILD ----- Go standard library only, no third-party dependencies, one source file. go build -o routewatch . Prebuilt binaries are in dist/: routewatch-linux-amd64 routewatch-darwin-amd64 routewatch-darwin-arm64 routewatch-windows-amd64.exe