=============================================================================== SystemPulse 1.0.0 - Command Latency Profiler Techlosoft "PC Performance Console" product line, Vertical variant =============================================================================== WHAT IT IS SystemPulse measures how long a SPECIFIC COMMAND takes to run, across many repetitions, and reports the whole distribution rather than a single average. It can save a run as a baseline and later re-measure the same command to answer the question a CPU benchmark cannot: "Did my app's startup get slower, by how much, and is that real or noise?" This is the differentiator from its siblings. CorePilot benchmarks raw CPU throughput; ThermalFlow benchmarks the CPU and keeps a persistent trend log. Both characterise the MACHINE in the abstract. SystemPulse characterises YOUR COMMAND on this machine, and turns that into a pass/fail regression gate. ------------------------------------------------------------------------------- INSTALL / BUILD ------------------------------------------------------------------------------- Go standard library only. No third-party dependencies, no network needed. go build -o systempulse . Prebuilt binaries are in dist/: systempulse-linux-amd64 systempulse-darwin-amd64 systempulse-darwin-arm64 systempulse-windows-amd64.exe ------------------------------------------------------------------------------- USAGE ------------------------------------------------------------------------------- systempulse bench --runs N [--warmup N] [--label NAME] [--save FILE] [--timeout DUR] [--json] -- [args...] systempulse compare --baseline FILE --runs N [--threshold-pct P] [--warmup N] [--timeout DUR] [--json] -- [args...] systempulse show --baseline FILE [--json] systempulse help | -h | --help EVERYTHING AFTER -- IS THE COMMAND UNDER TEST. Flags may be written before or after each other in any order; they are normalised before parsing. FLAGS --runs N Measured runs. Default 30. Must be >= 1. --warmup N Runs executed first and DISCARDED. Default 3. Must be >= 0. --label NAME Free-text label stored in the baseline file. --save FILE Write the bench result to FILE as a baseline (JSON). --baseline FILE Baseline JSON to read. --threshold-pct P Regression threshold, percent. Default 10. --timeout DUR Per-run wall-clock limit, e.g. 2s, 500ms, 1m. Default 0 = NO LIMIT. --json Machine-readable output including raw per-run samples. EXIT CODES 0 success / no regression 1 usage error or runtime error 2 regression detected above --threshold-pct Exit code 2 is what makes this usable as a CI gate. EXAMPLES # Profile a command and look at the distribution systempulse bench --runs 50 -- ./myapp --version # Record a baseline systempulse bench --runs 50 --label v1.4.0 --save base.json -- ./myapp --version # Later: fail the build if startup regressed more than 8% systempulse compare --baseline base.json --runs 50 --threshold-pct 8 \ -- ./myapp --version # Inspect a stored baseline systempulse show --baseline base.json ------------------------------------------------------------------------------- STATISTICAL DEFINITIONS (stated exactly, because they are checkable) ------------------------------------------------------------------------------- PERCENTILES: NEAREST-RANK. No interpolation. rank = ceil(p / 100 * n) evaluated in IEEE-754 double arithmetic rank clamped to [1, n] value = sorted_ascending[rank - 1] (1-based rank) Every reported percentile is therefore an ACTUAL OBSERVED SAMPLE, never an interpolated value between two samples. This is deliberately NOT numpy's default percentile, which interpolates linearly between neighbouring ranks. Comparing SystemPulse output against numpy's default WILL differ, and that difference is a definition mismatch, not a bug. Use the formula above. MEDIAN: defined as p50 under that same nearest-rank rule. For even n this is the LOWER of the two middle values, NOT their average. Example, n = 20: rank = ceil(0.5 * 20) = 10, so the median is the 10th smallest sample. Python's statistics.median() would average the 10th and 11th and give a different number. That is expected. STANDARD DEVIATION: SAMPLE standard deviation, Bessel-corrected. stddev = sqrt( sum((x_i - mean)^2) / (n - 1) ) 0 when n < 2 This is Python's statistics.stdev(), NOT statistics.pstdev(). MEAN and the sum of squared deviations are accumulated over the samples in their original EXECUTION order, so an independent reimplementation that walks samples_ns front to back reproduces the identical float64, bit for bit. WHAT IS TIMED: the full wall-clock duration of one spawn-and-wait cycle, measured around the process launch. It therefore INCLUDES fork/exec and process teardown overhead, which on Linux is roughly 1 ms per run. For a command that does real work this is negligible; for a command that does nothing it dominates. Child stdin/stdout/stderr are attached to /dev/null so that terminal I/O does not distort the measurement. EFFECT SIZE (the "is it real?" number): compare reports pooled_stddev = sqrt( ((n1-1)*s1^2 + (n2-1)*s2^2) / (n1+n2-2) ) effect_size = (median_current - median_baseline) / pooled_stddev |effect| < 1 -> "within run-to-run noise" 1 <= |effect| < 3 -> "moderate signal" |effect| >= 3 -> "clear signal" This is REPORTED ALONGSIDE the percentage so you can see whether a change is meaningful relative to the observed spread. IT DOES NOT CURRENTLY GATE THE PASS/FAIL VERDICT - see LIMITATIONS. ------------------------------------------------------------------------------- HOW THE PASS/FAIL VERDICT WORKS ------------------------------------------------------------------------------- compare computes the percentage change in the median and in p95, takes the WORSE (larger) of the two, and fails if it exceeds --threshold-pct. worst = max(median_change_pct, p95_change_pct) FAIL (exit 2) if worst > threshold_pct Improvements are negative percentages and always pass. ------------------------------------------------------------------------------- LIMITATIONS - WHAT IS *NOT* IMPLEMENTED ------------------------------------------------------------------------------- Read this section before trusting the tool with anything important. 1. WALL-CLOCK ONLY. SystemPulse measures elapsed real time of a process. It does NOT measure, and does not attempt to measure: - per-process CPU time (user/system split) - peak RSS or any memory figure - I/O counters, bytes read/written, syscall counts - system-wide CPU load, temperature, frequency or power All of those require OS-specific APIs (wait4/getrusage and /proc on Linux, proc_pid_rusage and mach APIs on macOS, GetProcessTimes and GetProcessMemoryInfo on Windows) plus per-platform code paths. NONE of that is implemented here. If a background process on your machine steals CPU during a run, that shows up in the wall-clock number and SystemPulse cannot tell you it happened. 2. THE VERDICT IS A RAW PERCENTAGE THRESHOLD, NOT A SIGNIFICANCE TEST. The effect size is reported but does not gate pass/fail. A change of 11% fails at the default threshold whether or not it is statistically distinguishable from noise. Proper hypothesis testing (Mann-Whitney U or a bootstrap confidence interval on the median shift) is roadmap, not shipped. 3. HIGH PERCENTILES NEED SAMPLES. Because nearest-rank picks a real sample, p95 collapses onto the MAXIMUM when ceil(0.95*n) == n, which is the case for any n <= 20. At n = 6, p95 IS the single worst run, so one scheduler hiccup moves it tens of percent and can trip the threshold on its own. The tool DETECTS this and prints an explicit warning in both bench and compare, but it does not refuse to run. Use --runs 20 or more for a trustworthy p95, and more than that for p99. 4. VERY FAST COMMANDS ARE INHERENTLY NOISY. Anything in the 1 ms range is dominated by process-spawn jitter, and run-to-run median swings of several percent are normal. Do not set a tight --threshold-pct on a command that does nothing. 5. NO PER-RUN TIMEOUT BY DEFAULT. --timeout defaults to 0, meaning NO LIMIT. With the default, A COMMAND THAT HANGS WILL HANG SYSTEMPULSE FOREVER. The protection exists but is opt-in: pass --timeout explicitly. When it is set, a run exceeding it is killed, counted in "timeouts" and in "failures", and its truncated duration is EXCLUDED from the distribution (a killed run's duration is not a measurement of anything). If every run times out, the tool errors out rather than reporting a fabricated distribution. Killing sends the signal to the direct child only; a child that spawns its own grandchildren may leave those behind. 6. FAILED RUNS ARE STILL TIMED. A run that exits non-zero took real time, so its duration IS included in the distribution, and its exit code is counted and reported. SystemPulse never silently treats a failure as a success, but it also does not exclude failures from the statistics. If your command fails fast, a rising failure count will make the command look FASTER. Always read the failures line. 7. NO SHELL. The command is executed directly, not through a shell. Pipes, redirection, globs and && are not interpreted. Wrap them yourself: -- sh -c 'a | b'. Note that this adds the shell's own startup to every measurement. 8. NO WARMUP OF THE OS PAGE CACHE beyond simply running the command --warmup times. There is no cache dropping, no CPU pinning, no frequency governor control, and no attempt to quiesce the machine. 9. BASELINES ARE NOT PORTABLE ACROSS MACHINES. A baseline records timings from the hardware and OS it was captured on. Comparing across different machines is meaningless. SystemPulse does not detect that you have done so. It does warn if the baseline's command text differs from the command measured now. ------------------------------------------------------------------------------- ROADMAP ------------------------------------------------------------------------------- - Per-run CPU time and peak RSS sampling via getrusage / proc_pid_rusage / GetProcessMemoryInfo, reported as their own distributions. - Statistical significance testing (Mann-Whitney U, bootstrap CI on the median shift) replacing the raw percentage threshold as the gate. - Flame-graph style breakdowns of where time goes inside a run. - CI integration: JUnit XML output, GitHub Actions annotations, and automatic baseline storage keyed by commit. ------------------------------------------------------------------------------- BASELINE FILE FORMAT ------------------------------------------------------------------------------- JSON, format_version 1. Notable fields: command argv of the command under test runs, warmup configuration used samples_ns raw per-run durations, nanoseconds, EXECUTION order stats count, min_ns, max_ns, mean_ns, median_ns, stddev_ns, p50_ns, p90_ns, p95_ns, p99_ns failures number of runs that exited non-zero or timed out timeouts number of runs killed by --timeout exit_codes map of exit code -> occurrences ("timeout" for kills) percentile_method the definition used, written into every file stddev_method the definition used, written into every file samples_ns is always present, so you can recompute every statistic yourself and check this tool's arithmetic. You are encouraged to. ===============================================================================